Potentially move away from actix

[bowenwang1996]

We have been using actix for quite some time now, and have encountered several issues during this time period. Recently, after the actix upgrade (#3869), issues like #3925 appear and prompted us to reconsider whether using actix is a good idea after all. I'll attempt to list the advantages and disadvantages below to hopefully provide a holistic view of the situation:

Advantages of keeping actix

• Nice API and very easy to use.
• We have already been using it for almost two years and it has integrated deeply into our codebase. If we were to change this, it will require a significant amount of work.

Disadvantages of keeping actix

• There is a maintenance cost: we need to keep an eye on the actix version updates and bugfixes.
• Issues like CI failure after upgrading actix #3925 indicate that actix may have some design issues.
• Debugging actix code could be annoying, even if the bug is not inside actix, as @pmnoxx found during his investigation of Very high idle CPU usage due to peer Arbiter threads #3854.
• Actix does not give us the flexibility to customize message delivery. For example we may want to prioritize certain messages over others in the future.
• We may not actually need a full-blown actor framework like actix, as @matklad suggested here and could potentially build something much simpler on tokio directly.

I would love to see whether people think we should keep using actix. @SkidanovAlex @frol @pmnoxx @matklad

[Kouprin]

Can you please create a short list of issues caused by actix? I quickly found #2606 and #2178.

[bowenwang1996]

Some recent issues include #3925, #3930, #3854. Some of these are not directly issues with actix, but are caused by bugs in actix dependencies or change in their APIs. I am sure we had more issues in the past and I think @frol and @matklad provided a better overview on this topic.

[pmnoxx]

@bowenwang1996 That's true. However, having Actix adds one extra layer of code, we need to update Actix at the same time as dependencies. It would be safer to update one dependency at at time.

[frol]

@pmnoxx The dependencies issue is mostly caused by the fast pace of iterations in async-ecosystem. I believe it is going to settle down given the Tokio 1.x release. Streaming API is still under design, and futures still need 1.x release. I feel that we have more pressing issues than implementing our own framework, but if you feel otherwise I am fine with that.

[pmnoxx]

@frol IMHO From business perspective, in short term, it makes sense to focus now on improving our protocol first, and fixing other issues. For example, planning on how to change the protocol to scale beyond 300 nodes would be far more useful than replacing Actix. For example, ClientActor should be completely rewritten. At the very least we should wait for futures to get 1.x release.

In the long term, It makes sense to switch away from Actix, and implement a custom framework that should be relatively simple and easy to main, while also would add features we need.

I can dedicate a week writing/investigating a prototype framework and see how it goes. It should be possible to write something that can be easily extended and if needed finished by 3rd party under our supervision. IMHO It's a low priority compared to other things we can do to improve Near, like working on networking, etc.

Though, I agree with @bowenwang1996. Right now we have Actix, running on top of Tokio, which runs on top of Mio. Debugging is harder, because you have to go through many layers of abstraction to find issues.

[bowenwang1996]

@frol yeah I agree that it is not a priority for now, but it doesn't mean we shouldn't start thinking about it and discussing potential alternatives.

[frol]

There is a maintenance cost: we need to keep an eye on the actix version updates and bugfixes.

Well, there are a few things:

• after actix shifted into the hands of the community, it has gotten some better maintenance, and we had only finally migrated to the community-released version of actix with the recent dependencies upgrade
• the whole async ecosystem has been in flux before Tokio 1.0 release, and now it is going to settle down with most of the packages gradually being upgraded to Tokio 1.0

Thus, I think it will be less of an issue in the future.

[matklad]

Clarification to avoid confusion: this discussion is about actix the actor framework, not the actix-web http framework.

[matklad]

Preface: I have experience with concurrent synchronous Rust. I don't have a
lot of "production" exp with async world. In particular, rust-analyzer works a
bit like a blockchain node might: there's network layer, which talks JSON-RPC,
there's state of the world which is managed by central "actor", and there's a
thread-pool to execute long-running CPU-heavy computations. But, as the number
of external processes we communicate with is low (1-10), we use simple
"thread-per-connection" model.

I think that "managing concurrency in a stateful peer node" is a core competence
of NEAR. That's what replicated state machines are about, after all (in
contrast, I don't think HTTP is a core competence). So imo it makes sense to
have deep in-house expertise about how exactly concurrency is handled in near
codebase.

I also believe that concurrency is hard. It's easy to arrive at "works most of
the time" solution, but ensuring that it "works in the worst case" is
challenging. I wish there was a framework that handled the hard parts for us.

My cursory look at actix actor framework makes me believe that it doesn't tackle
hard problems for us.

The first hard concurrent problem is termination. You want to ask concurrent
system to shutdown, and be sure, when this call returns, that the system is
fully down. This doesn't appear to be the case with actix -- the actor can
outlive it's System. That's the root cause of #3925. The minimal demonstration
of the problem is this
gist.

A similar issue is cancellation. It should be possible to cancel some work, and
wait until the cancellation is acknowledged and the work has actually stopped.
Actix stop method don't seem to handle the and wait part.

The second hard problem is backpressure. Actix uses fancy channels for
mailboxes, which do handle backpressure in some fancier-than-usual way (the
specifics are not documented, and unclear from a cursory glance at the code).
The problem is, if the actor is async (handle uses spawn or return a
future), the mailbox is bypassed, the future is added directly to the actor's
run queue, and backpressure isn't applied. Here's a
demo

At this point, I am not sure what, exactly is being handled by actix. A
System feels like a wrapper for tokio's Runtime. An Arbiter is a wrapper
around std::thread. An Addr is a wrapper around channel (here I see
potential value added in terms of custom backpressure strategy, but, again, I am
not sure what that strategy is exactly). Most annoyingly, ActorFuture is a
wrapper over Future, which has its own set of combinators, and thus doesn't
work with .await. All this uses fancy type-system trickery, like curiously
recursive associated types, which adds significant cognitive load when you try
to understand what exactly happens and when.

Contrast this with a minimal
manual
implementation of actor, where it is crystal clear what the system is doing.

All this being said, I also agree with @frol that a big part of accidental
complexity here is that we are playing Achilles and Tortoise with the async
ecosystem, and are using pre-release versions of the crates. I think the best
course forward is to wait until we can get off betas, and then spend some time
to experimentally port some bit to be actix-less, just to have more meaningful
comparison between two versions of code.

[pmnoxx]

Thoughs about async rust

I have written production code with async Rust in the past.It has some good sides, but also some bad sites.
There are a few limitations, which make it hard, but not impossible to use:

• Last time I checked, generics/traints can't use async, this mades it harder to write a clean code.
  I like the way actix implemented introduced Actor, adding a new method, simply means implementing new Handle for given Actor

2. You can't use mutable arguments as parameters to async functions, that would break mutable reference rule, if you spawn more than one future, as each needs to keep reference.
   You can do a workaround by using Arc<RefCell<Self>>
   However the code would look like this:

async fn example(Arc<RefCell<Self>> self) {
    self.map(s || {
       ...
    })
    do_call(self).await;
    self.map(s || {
       ...
    })
}

3. Also it's a little bit harder to debug

We would most likely have to use Actors like we currently do. However, I think we should chang how handling actors is implemented.

Main reasons for a change

There are a few issues with actors we need to solve

1. Lack of performance metrics
   This can be implemented inside Actix, though it would be easier to simply write our own Arbiter.

2. Lack of prioritization.
   Currently on a arbiter thread, one of few thing can ran
   a) Actor - handle method
   b) network read/write
   c) call_later code
   We don't control what gets executed first, that can lead to a couple of issues.
   For us controlling flow of messages is a high priority, controlling proritization is simply now enough.

3. Lack of backpressure.
   We need to be able to protect ourselves against situations, where some peers send more trafic than others.
   Also, we need to avoid a situation where a Peer sends data faster than we can process.
   Each message in our system should have assosiated Peer with it. This would allow us to limit the number of work a Peer can cause.
   4)Tracing down which futures are still alive or not,
   I've seen cases, where we would schedule thousands Routing Graph recomputations, even though we didn't finish previous one. (change peer_manager not to recompute routing multiple times #3891)

4. Code complexity, we don't need many features, which actix provides, we could write a simpler layer between us and Tokio.

State of Actix

We started using Tokio/Actix, before they were ready for production.
We've experienced a few technical issues, I'm sure they will get fixed, and we will start experiencing less issues.
Some features can be imlpemented in actix, for example performance tracing,
We need to control how backpressure is implemented, so that would be hard.

How this can be done

The solution to these problems will not be as simple as replacing Actix alone.
Actix is a wrapper around Tokio, Actix adds more work to Tokio using tokio::spawn.
tokio::spawn this method doesn't provide any mechanisms for proritiazations, adding backpressure.
Most likely we would be required to either stop using Tokio, and implement our solution on top of mio.
That can be done, I've experience of writing my own Json Rpc server in Rust directly on top of mio.

We can probably keep using tokio as well, if we implement an Arbiter as a future.
We would spawn exactly one of such future on a tread using tokio::spawn.

Mailboxes,

currently mailboxes are unlimited in Actix or they block current thread. It would be a good idea for spend some time thinking of an alternative.

[ailisp]

I checked solana and polkadot, both of them use tokio directly and no actix. Haven't taken a deeper look on whether and how they handle metrics/mailbox/prioritization/back pressure

[pmnoxx]

About prioritization

I thought for a while about how we can add prioritization/back pressure. Whenever it's possible to do it on top of Actix/Tokio.

It's safe to assume we want to continue using some kind of IO loop, which schedules which futures to run.
That includes processing messages by given Actors and also doing IO like reading/writing from/to a socket.

Each Future in Rust is associated, which a waker, that's an object that gets informed when future is ready to be executed. For example, whenever socket is open for reading/writing, Actor has messages to which are not yet processed.

Tokio::spawn(...) method which Actix uses, adds a future to list of futures which are processed by Tokio's IO loop. There is no prioritization, so if we want to be able to add prioritization we need to get rid of both Tokio and Actix for handling IO loop.
We should probably rename this issue to be "Potentially move away from Tokio".

I wrote a custom RPC server in Rust in the past on top of MIO with support for backpressure/prioritization. So I have experience doing it, but it was never published.

To implement prioritization I would do the following, give writing IO the highest priority,
each message would have it's priority between highest and lower, and lowest priority would be reading data from socket.
The IO loop would look like this:

// Code for a single Arbiter thread:

timeout = 0;
// select returns whenever new events happened or timeout passed
while new_events = MIO::select(timeout) {
  // we can do prioritization like in linux, we have 40 priorities from 0..40
  // for each priority in 0..40 we keep a queue of futures, that need to be executed
   add new_events to our data structure
   choose element with highest priority
   execute that future
   
   if there are event lets to do {
       timeout = 0;
   } else { 
       timeout = 0.01;
  }
}

With a careful implementation, we can add a back pressure as well, but that should be a topic for another discussion.

[pmnoxx]

About backpressure

Adding back-pressure makes sense when we think about Peer as a mini-network of Arbiters. Each Arbiter represents a single threaded peer or thread pool of peers.

You have Arbiter with ClientActor, one for peer, one for EdgeVerifier, one for ViewClient, etc.
Let's say our current node is connected to peers p_1, p_2, ... p_k. We would like to stop reading messages from a peer if peer is sending too much data, and we can't process it.

A peer send send 2 types of messages:

1. A routing message, from p_i to p_j:
   If Peer actor j has too many messages scheduled to j exceeding memory limit, we can throw away such message. If we were to stop reading from p_i completely, that could cause an issue, when if p_j is slow, we stop communicating with p_i.

2. A message send directly to p_i that needs to be processed.
   We could for example keep a global counter of all messages p_i has send to us, but were not processed yet, across the node.
   That message could be for example a block chunk, not yet processed by ClientActor.
   In that case in PeerActor i, we would suspend the future, responsible for reading from the socket, until we have capacity to process more messages.

Using a back pressure as implemented above, we could protect Node, against another peer sending too many messages, bringing down node due to high CPU usage / OOM.

[matklad]

We should probably rename this issue to be "Potentially move away from Tokio".

@pmnoxx my understanding is that this is explicitly not what this discussion is about: #3962 (comment)

[pmnoxx]

@matklad This is related, my point is that we can't add prioritization and backpressure by only removing Actix. So some strong points of switching away from Actix are no longer valid.

[pmnoxx]

Switching away from Actix in a simple way

It's safe to assume for now that we will not be able to add prioritization/back-pressure by simply switching away from Actix.
We need to have a plan which will allow us to move away from Tokio eventually, but can be executed in small steps.
I propose we do the following.

Step 1) (For now)
We should be able to implement our own Arbiter/SyncArbiter, first by extracting appropriate code from Actix, then rewriting it. I'm in favor of using Actor system as it is with inboxes, though we would be able to control this code.
Step 2) (In the future - optional)
Replace Tokio, with our own IO loop.

Advantages

• Control shutdown process, fix issues we experience
• We can add our metrics code, easily measure how much time executing each future takes, how big are queue, etc.
• Add debug code to find more potential issues
• We get to keep nice API
• This can be a good step forwards removing Tokio in the future, it will allow us to add prioritization/back-pressure
• It should be possible to convert existing Actors to new ones on at a time if needed. We would keep initially API close to what Actix has, that should require minimal changes to our code.

Neutral

• We will not get rid of Actix/Tokio all at once, this migration can be split into small steps, that should be easily testable.

Disadvantages

• We will end up creating our own layer similar to Actix, but simpler. We will have to maintain it, that should be ok. We can extend the score of the project in the future, and create even better library.

[matklad]

https://uazu.github.io/stakker/d-intro.html (toc in the sidebar) is an interesting read regarding the design of actor systems in Rust.

[frol]

I just bumped into an actor framework I haven't seen before: https://github.com/rillrate/meio (the author has also brought Yew frontend framework to life)

It claims to have a concept of high-priority tasks, graceful termination on sub-actor and task level.