How does "runtime-agnostic" work?

[gokayokyay]

Hello everyone!

First of all, thanks for the amazing library, it's a pleasure to use for newbies like me!

Let's begin with a little story. I've been using isahc in a project. Actually I was going to use reqwest because I've already integrated tokio but I got lot's of incompatibility issues with tokio versions.

So I found isahc and saw that it's runtime-agnostic and then I thought awesome, now I'm gonna use it. That happened over 1 month ago. Today I was continuing on the project (it's kind of a hobby/side-project so I don't have any rush) and just realized that isahc, the http client which relies on curl, is runtime-agnostic! Since then I was like what the hell how is isahc runtime agnostic? How did they achieve that?

I've been trying to deep dive into async stuff. Like how does that work or what makes something async etc... So far I learned that tokio or async std relies on the system calls and polling like epoll etc... But how a wrapper (or a binding) can be async? When I digged down the code, I saw the "interceptors". So my main question will be, how do the interceptors work?

Thanks a lot!

[sagebind]

Thanks for asking, I like answering questions like this! This is a big topic and it seems like there are multiple questions here, so let me try and split it up and respond to each:

• How does async work in general and in Rust?
• How does async work in Isahc?

This response will be rather long, so strap in! This is a topic that has been on my todo list to write about, so I'll probably extract and expand this into a full article in the future (no pun intended). I don't know much systems programming background you have, so I'll try to summarize various levels just in case you (or future readers) are not familiar with them.

---

What do we mean by "async"?

Firstly, getting into asynchronous (async) programming can be a bit challenging at times, because we often use the same word to mean multiple different things. At its core, it simply means that two "parties" are operating together in a manner that is not synchronous; that is, not always well-ordered in sequence.

What these parties are and how this is done all depends on the context. For example, in programming languages we are usually talking about things like asynchronous I/O, but in a distributed system we might simply mean that two systems are not making synchronous calls to each other and may operate mostly independently. In this context, how programming languages work or how I/O is done is irrelevant, and we're more interested in whether one party "waits" for responses after sending a request, or instead it continues with its work and listens for an "asynchronous" notification of a response instead.

This is a pretty large, broad topic, so I'll avoid getting into too much detail with this first part. You could probably fill a whole book talking about this (something I've thought about doing) and it won't fit here.

Non-blocking operations

Now in the context of programming languages, async typically refers to non-blocking operations, which like asynchronous is defined in terms of what it isn't. A blocking operation is pretty simply any operation or instruction that will prevent some thread or process from making forward progress until some other "thing" that is responsible for actually making the operation happen is finished. (There's probably some academic disagreement on an exact definition!) A non-blocking operation then is something that, well, doesn't do that.

Let's try to look at a realistic example: sending a packet of data over a socket to another computer. In a normal, blocking program, you might simply call a function such as send() or write(), which will invoke some system calls that the operating system will handle for you. The operating system then will send the data on your behalf. Except there's a problem: the Ethernet cable isn't soldered directly to the CPU! There's nothing you can do with a CPU alone to send that packet! So how does this work?

The way this is accomplished is by interacting with another piece of hardware that is connected to the network (through various layers of drivers and hardware), usually a network interface controller (NIC). Now the NIC doesn't really need the CPUs help in doing networky things, it can sit there and do its own thing, but it does need to be told what to send and receive from the network.

So in our case, the CPU gives the NIC the data we wanted to send over the network. But now we have another problem: what if the NIC is busy doing something else right now? This is where blocking will come in. The operating system will queue up the request to send data somewhere, and then put our program to "sleep" until it is our turn to use the NIC, the NIC does the operation, and the NIC interrupts the CPU informing that the operation is complete.

Remember, the CPU is free to do anything else it wants while the NIC is doing this, but our program is not because of the way we wrote it; we expect the lines to be executed one after the other, so we cannot continue until our send() call finishes. Now, this isn't exactly a problem though. For a simple program, we probably don't have anything better to do anyway while we wait for the NIC to send our packet. The operating system putting our program to sleep, or blocking, is a feature!

But... what if we do have something better to do than just twiddle our thumbs? I mean, we could create more threads and processes that could all sleep simultaneously, but those are relatively heavyweight operating system resources if we don't really need them. Creating a new thread and allocating a new stack seems silly if it is just going to be asleep 99% of the time and offers no other benefit.

This is where non-blocking operations become useful! Instead of calling send() and then sleeping, we could use something theoretically called send_nonblocking(), which when called, the operating system will send things over to the NIC as usual. But instead of putting our process to sleep, it just returns immediately saying, "Hey, I submitted your request, it will be done sometime later." This is what Windows does, or Unix-like operating systems will return saying, "Hey, I can't do this immediately right now, try again in a bit."

This is great! Now we can start doing other useful things with our time instead of just twiddling our thumbs! But now we have a new problem: When is our operation done?

Readiness and completion

In order to figure out when things are done (or for Unix I/O, when things can be performed immediately without blocking) we need to use OS-specific tools like epoll, select, IOCP, etc. These all work in a little bit different ways, but the core idea is typically the same: on one thread, you can block on special system calls that will put your thread to sleep until one or more operations are ready. Sure, you are still blocking (it is almost impossible to get away from this), but instead of needing to block N threads in order to perform N operations, you only need to block 1 thread for N operations. The OS will wake you up when any of the operations are ready and you can proceed from there.

Now this is pretty handy and lets you be pretty efficient about things, but is pretty unwieldy to use and can lead to some pretty messy code with massive switch statements in a big loop. So how do we deal with that?

Rust's async/await

Now we get to the Rust specific part. Some languages like C have no answer for the above problem, but Rust does! This is what futures and async/await is for: it is an abstraction layer that takes the complexity of this stuff and moves it somewhere else, so that you can write your programs in a relatively sane way again, but still getting the benefits of non-blocking operations as before.

Now it is helpful to think of Rust's futures as a building block for working with non-blocking operations. They don't do anything themselves, they are simply a building block, a common interface. To actually implement all this, you're going to have to implement something like that big loop from before somewhere.

A common solution is to provide something called a "runtime", which is simply a library that knows how to listen for many operations at once using the appropriate OS calls. Tokio, for example, will have multiple threads running a loop like this (potentially your main thread too) and whenever an operation completes, advances a block of code (or future) to the next steps until either it is complete or it makes another async call. In this way, Tokio is kind of acting like an OS, by putting your asynchronous code to "sleep" until your .await call is complete, except it can do this using just one or a few threads. While one future is "asleep", another can be "awake" on the same thread. (Of course, only one future can be actively running on one thread at any given time, but an unlimited number could be waiting at a time.)

How is Isahc async?

Now to use all this, one needs to write your code with specific knowledge of Tokio. You need to tell Tokio what sort of I/O you are doing and how, so that Tokio knows how to handle all of it in that big loop. But there are other ways of approaching this problem as well, which I'm going to call micro-runtimes. In this model, you essentially implement a runtime that knows how to do exactly 1 thing for 1 specific purpose.

Let's use timers for example. You could create an asynchronous timer library that uses its own micro-runtime by maintaining exactly 1 background thread. On that thread, you could sleep until the soonest timer is ready. Once ready, you wake up any and all futures that were waiting on it (this is what Wakers are used for), then repeat the process for the next timer. Since this design doesn't rely on the behavior of a central loop (which is called an event loop, a term you may have heard), it is naturally runtime-agnostic in a sense. It can be used with any other code that uses any other runtime, as long as you rely exclusively on the common interface of futures.

This is how Isahc is implemented. Isahc essentially ships with its own internal runtime called an agent, which runs in a background thread and drives a loop that dispatches curl handles in a non-blocking way and waits for socket activity on all of them simultaneously. Since Isahc calls always use an Isahc agent, it does not depend on the caller to use any particular runtime themselves.

Are there any disadvantages to these micro-runtimes? There certainly can be! Using background threads, even just one, is still one more thread in your program you might otherwise not have needed. This is not usually an issue, but can increase the amount of memory used, and won't work at all in a program where you either can't or won't use multiple threads (e.g. WASM, embedded, etc). For Isahc and the sort of use-cases I want it to meet, I think that the tradeoff makes sense, but for other things it might not.

Note that some asynchronous libraries can be implemented without any runtime like this at all, such as an asynchronous channel. There you can use simpler algorithms such as having the sender wake up futures waiting on the receiver in-line after the user sends a message, the receiver waking up futures waiting on the sender, etc. In this case there's usually no tradeoff to make.

I'd also like to add that this isn't the only way to make a runtime-agnostic async library. Another approach would be to use compile-time features to use specific runtimes. Another approach would be to introduce a common interface for various operations that all runtimes would implement, but I don't know that this will happen except for specific operations such as spawn.

---

I know this was a lot of information, and not quite as organized as I'd like it to be, but hopefully these answer your questions! If not feel free to ask anytime.

Thanks for using Isahc!

[sagebind]

So I wrote all of that and forgot to answer your final question!

When I digged down the code, I saw the "interceptors". So my main question will be, how do the interceptors work?

Interceptors in Isahc have nothing to do with implementing async in Isahc; the place for that would be to start in agent.rs. Interceptors are just middleware layers that let you customize the behavior of Isahc. Interceptors have to themselves be async because the rest of Isahc internally is async, even if you were to use the synchronous API.

[gokayokyay]

I didn't expect this long reply! I really can't thank you enough, you've answered almost all of my questions (and some of my future questions too actually 😀). I really think that this post should be in every async rust guide's (or framework's) first page, and I mean it. It would be much more easier if someone had explained it like this.

There are some points that I want to make clear though:

1. So Tokio actually somehow wraps blocking system calls and since it wrapped them itself, it knows when to wake the calls too.
2. If I want to use only one thread, I could make some system calls and group them in a one big loop and a queue then poll each of them until every single one becomes ready.

You've mentioned the asynchronous channel, I've been seeing it everywhere where everyone says you can wake up future without using an extra thread, but they didn't provide any examples. And if there is usually no tradeoffs to make, why did you choose the path to make the micro-runtimes (it sticks to tongue 😀).

Again, thanks a lot! I seriously can't express how this post made me relieved 😀. I can't wait to read your blog post too!

[sagebind]

So Tokio actually somehow wraps blocking system calls and since it wrapped them itself, it knows when to wake the calls too.

Yes and no. Tokio uses non-blocking variants of most operations if they exist. On Linux, this just means setting the O_NONBLOCK flag, and then reads and writes will never block. Again, this is OS-specific on how to do this.

To know when to wake futures, Tokio is blocking a limited number of threads with a separate call, somewhat unrelated to the actual operation, to be alerted by the OS whenever an operation will either not block, or is finished. The idea here is you probably cannot avoid blocking somewhere, but here we can limit it to maybe 1-4 threads that can listen for readiness of thousands or more concurrent operations, instead of blocking thousands of threads individually.

If I want to use only one thread, I could make some system calls and group them in a one big loop and a queue then poll each of them until every single one becomes ready.

Exactly! The idea is to minimize and isolate where blocking occurs, and to get as much "bang for your buck" as possible when you do block. You want to be careful though; you don't want to "poll each of them" individually, but rather you want to use a system call or other strategy that allows you to poll all of them simultaneously. One aphorism commonly used is, "Threads are for executing in parallel, async is for waiting in parallel."

[gokayokyay]

Yes and no. Tokio uses non-blocking variants of most operations if they exist. On Linux, this just means setting the O_NONBLOCK flag, and then reads and writes will never block. Again, this is OS-specific on how to do this.

Oh it became much more clear now. When you do not have such a syscall it's better to use some helper thread to wake up 😄. Thanks again, you really helped me in my way on learning async then async rust 😄.