Consider supporting advanced networking APIs

[Ralith]

Most Unixes today implement the RFC2292 Advanced Sockets API to allow ancillary data to be sent/received alongside datagrams using different control messages. Examples include explicit congestion notification flags and hardware packet timestamps on UDP/IP sockets, and file descriptors on Unix sockets.

Is tokio interested in providing direct support for these use cases? The API is idiosyncratic, requiring large amounts of aggressively unsafe code (see e.g. the draft implementation of ECN handling in quinn), so a safe binding would be valuable. On the other hand, platform support for specific control messages varies, and libc is currently missing bindings for the macros involved in encoding/decoding control messages on all platforms but Linux.

Such an API would allow any write operation on a datagram socket to be supplied with a stack-allocated buffer of control messages constructed in a memory- and type-safe manner, and any read operation supplied with a stack-allocated buffer to which it can write arbitrary control messages, which can be safely iterated over on success. Accomplishing this with minimal overhead wouldn't be trivial. For example, we could define UdpSocket::poll_send_to_advanced which takes a slice of enum ControlMessage { ... } values, but this would require conversion internally, perhaps using a SmallVec to avoid the heap most of the time. Returning a set of control messages has similar challenges.

[carllerche]

I definitely want to support a way to use these advanced APIs. The question is going to be what that way is :).

I'm not sure what the latest and greatest way to provide platform specific APIs is. Is an extension trait still the strategy?

[Ralith]

@aturon's post on portability seems relevant. He advocates for just cfging things out and letting a (not yet implemented?) lint handle potential surprises.

[carllerche]

@Ralith sounds good to me! Looking forward to any PRs 👍

[pusateri]

In nix-rust/nix#990, I recently added IP_PKTINFO & IP6_PKTINFO ancillary data cmsg support with recvmsg() and sendmsg(). Then in nix-rust/nix#1002, I'm adding IP_RECVIF and IP_RECVDSTADDR (IP_SENDSRCADDR). I would like to see this get migrated to socket2 and eventually into std.

Other ancillary data can be added there easily. If it was easy to swap in a different underlying socket implementation, this could be a pathway.

[Ralith]

Nix's approach seems to be unsound (per nix-rust/nix#999), and the CmsgSpace trick, while clever, is pretty ugly. I'm not sure what a better API would look like, though. The ABIs here are practically a diverse set of ever so slightly incompatible binary network protocols. Maybe something flyweight-y that doesn't expose references?

[pusateri]

I used nix because it was easy to add additional control messages to but I am flexible and ultimately, just need a solution for tokio (preferably with async/await). I tested nix on lots of operating systems and architectures including actual hardware amd64, armv7, powerpc, & mips64 using *BSD and Linux variants. So while there may be a theoretical alignment issue with certain control messages on certain architectures, I never was able to hit it. But I don’t mind fixing it if it can be reproduced.

[asomers]

You're damn right that Nix's cmsg code is ugly! But have you ever looked at cmsg code written in C? It's ugly too. The entire sendmsg/recvmsg API is very cumbersome. If you have a better idea about how to Rustify it, please raise an issue in Nix!

[Ralith]

But have you ever looked at cmsg code written in C?

Yes, I'm currently using C-style code in Quinn. It's not great either.

If you have a better idea about how to Rustify it, please raise an issue in Nix!

It's a hard problem, for sure. Time permitting, I plan to experiment with a flyweight-y by-value approach, sort of like capnp (de)serializing if you've ever used that.

[carllerche]

I'm fine adding more APIs, however I'm going to close this issue as there is no immediate bandwidth to implement this.

If someone wishes to champion this, we can re-open it or they can discuss in Gitter and submit a PR.

[ghenry]

Hi all,

Did this just die off? I'm looking to use IP_PKTINFO etc with Tokio.

Thanks.

[Ralith]

I think everyone who's had a go at this has determined that it's really hard to design a good API to the mess of platform-specific ad-hoc interfaces that is ancillary data, and that using the platform interface directly isn't that bad. quinn-udp is seeing some use outside of Quinn as an ugly-but-simple abstraction for specifically the stuff that QUIC needs.

[Darksonn]

Yep. For now, the answer is to use socket2, nix, or libc to make the syscall directly within Tokio's try_io utility.

[ghenry]

Thanks Alice. Do you know of any examples anywhere?

[Darksonn]

Sorry, I don't have an example on hand.

[siennathesane]

Here's a advanced networking API example for "hardware timestamping" on macOS.¹

use std::{
    io,
    os::fd::{
        AsRawFd,
        FromRawFd,
    },
};
use std::net::{IpAddr, Ipv6Addr, SocketAddr};
use libc::SO_TIMESTAMP;
use socket2::{Domain, Socket, Type};
use tokio::net::TcpSocket;

#[tokio::main]
async fn main() {
    let socket = Socket::new(Domain::IPV6, Type::STREAM, None).unwrap();
    let fd = socket.as_raw_fd();

    let enable: i32 = 1;

    unsafe {
        if libc::setsockopt(
            fd,
            libc::SOL_SOCKET,
            SO_TIMESTAMP,
            &enable as *const _ as *const libc::c_void,
            size_of::<i32>() as libc::socklen_t,
        ) < 0
        {
            panic!("setsockopt failed: {}", io::Error::last_os_error());
        }
    }

    let addr = SocketAddr::new(IpAddr::V6(Ipv6Addr::new(0,0,0,0,0,0,0,1)), 8080);
    let tcp_socket = unsafe { TcpSocket::from_raw_fd(fd) };
    tcp_socket.bind(addr).unwrap();

    let listener = tcp_socket.listen(1024).unwrap();

    loop {
        let (stream, _) = listener.accept().await.unwrap();
        println!("Accepted connection from {:?}", stream.peer_addr().unwrap());

        // Buffer to receive actual data
        let mut buffer = [0u8; 1024];

        let mut ts = libc::timeval {
            tv_sec: 0,
            tv_usec: 0,
        };

        // Control message buffer
        let mut cmsg_buffer = [0u8; 128];

        // Setup iovec with our actual receive buffer
        let mut iov = libc::iovec {
            iov_base: buffer.as_mut_ptr() as *mut libc::c_void,
            iov_len: buffer.len(),
        };

        // Message header
        let mut msg = libc::msghdr {
            msg_name: std::ptr::null_mut(),
            msg_namelen: 0,
            msg_iov: &mut iov as *mut _,
            msg_iovlen: 1,
            msg_control: cmsg_buffer.as_mut_ptr() as *mut _,
            msg_controllen: cmsg_buffer.len() as libc::socklen_t,
            msg_flags: 0,
        };

        unsafe {
            // wait for actual data
            let ret = libc::recvmsg(stream.as_raw_fd(), &mut msg, 0); 
            if ret < 0 {
                eprintln!("recvmsg failed: {}", io::Error::last_os_error());
                continue;
            }

            println!("Received {} bytes of data", ret);

            let mut cmsg = libc::CMSG_FIRSTHDR(&msg);
            while !cmsg.is_null() {
                if (*cmsg).cmsg_level == libc::SOL_SOCKET && (*cmsg).cmsg_type == SO_TIMESTAMP {
                    let ts_ptr = libc::CMSG_DATA(cmsg) as *const libc::timeval;
                    ts = *ts_ptr;
                    break;
                }
                cmsg = libc::CMSG_NXTHDR(&msg, cmsg);
            }
        }

        println!("Timestamp: {}.{:06}", ts.tv_sec, ts.tv_usec);
    }
}

Footnotes

1. Ironically, while this code compiles and runs without error, xnu gives you 0 actual control messages on TCP sockets, but this is just an example so it's fine 😅 ↩

[Darksonn]

That example is wrong.

// wait for actual data
let ret = libc::recvmsg(stream.as_raw_fd(), &mut msg, 0); 

There's no .await here, so you're not waiting for data asynchronously. You must use TcpStream::try_io to implement something like this correctly.

[siennathesane]

That example is wrong.

Someone asked for an example, so I wrote up a quick example 🤷🏼‍♀️ It does work on Linux if you don't care about the packet payload, which I didn't for the purpose of the example since it was to demonstrate how to get hardware timestamps from the control messages.

Feel free to provide a correct example.

[Darksonn]

No, it does not. Using this with Tokio will not work correctly. I'm not talking about the payload being missing. Tokio keeps track of whether IO resources have data available, and you are not updating Tokio's tracking of readiness.

[ghenry]

Thanks all. I'll be back on this in a few weeks and will have a play. Will share any examples I have. I've bumped it down a few chapters in my book whilst I work on something in it.