feat: VM Execution Lanes

[vyzo]

Overview

This pr implements effective execution lanes for the fvm, supporting a default and a priority execution lane.

The priority execution lane reserves a number of execution units so that there is always some capacity for it even if default execution load spikes, typically through user load.

Implementation Notes

• Configuration is handled through the LOTUS_FVM_CONCURRENCY (also used by the fvm itself to reserve execution units) and LOTUS_FVM_CONCURRENCY_RESERVED env vars.
• Currently, only ApplyBlocks execution is prioritized, as it is critical for sync/consensus.

[raulk]

As we discussed synchronously, we need to justify this number by looking at the critical path usages in the chain sync subsystem. From my understanding, those are the following, but @magik6k will probably have better ideas.

• Incoming block validation via gossip
• Head coalescer executions.
• Actual deferred execution.

Also note that time-sensitive miner gas estimations and mpool pushes will fall under the "user" section. I'm not sure how much of a problem that could be, probably not much since those daemons won't be facing arbitrary user workloads. But worth confirming.

[raulk]

I haven't reviewed this whole file yet. Pending.

[arajasek]

Broadly looks like it should work, some notes.

[fridrik01]

Looks good, just some minor comments

[fridrik01]

Can we add a comment somewhere that DefaultAvailableExecutionLanes include the DefaultPriorityExecutionLanes so they are not added together (unless I am somehow terribly mistaken).

[vyzo]

ok, this should not be confusing.

[fridrik01]

So I understand, is the executionToken.reserved mainly used to handle the case where we have a ExecutionLanePriority where all the priority lanes are used/full but there are free default ones?

[vyzo]

exactly, it is the spill mechanism.

[arajasek]

LGTM, but let's let @fridrik01 also approve.

[fridrik01]

LGTM also

[Stebalien]

Doing this entirely in lotus is looking pretty gnarly. Someone is going to forget to call Done in some error path and cause a deadlock (or call Done multiple times).

Please don't merge this yet, I think there's a better way and I'll spend a few minutes trying to flesh it out.

[Stebalien]

Ok, so:

1. The benefit of this approach is that it limits the lotus node to LOTUS_FVM_CONCURRENCY concurrent machines, limiting memory usage.
2. The downsides are:
   1. This prevents interleaving message execution. This doesn't matter for throughput, but it increases the latency of otherwise fast requests by forcing them to block on other slower requests.
   2. It infects everything with .Done() statements.

An alternative approach is to handle this at the ApplyMessage level. I.e.:

1. ApplyBlocks calls ApplyMessage with a priority of 0.
2. Everything else calls ApplyMessage with the default priority (e.g., 100).
3. Internally, ApplyMessage uses a semaphore to stop "normal" priority calls when the number of available lanes reaches LOTUS_FVM_CONCURRENCY_RESERVED.

Unlike the current approach, this doesn't require any deferred calls to some "done" method. Really, this can be entirely handled with a flag in the context passed to ApplyMessage.

In the future, this can be refined further: Instead of always reserving a couple of lanes, ApplyBlocks can raise the number of reserved priority lanes when called and reduce them when it returns.

[vyzo]

calling Done multiple times is idempotent, so that part is safe.

[vyzo]

Your proposal would use a similar mechanism (ie tokens for execution lanes).

The advantage it has is that the lock is more finegrained and that the Done call is limited in scope at ApplyMessage.

[vyzo]

I am not opposed to doing it that way, but the benefit really is marginal imo.

[vyzo]

Also, we dont need the context option at all, we can do the locking at the ApplyMessage shim and have the priority come from vmopts as it currently does.

In short, we can hide the whole getToken/putToken/Done business inside the ApplyMessage shim of the executor, and it's not the end of the world if you forget to call Done outside.

That's a change I am more willing to implement.

[vyzo]

See #10590 for implementation of the finegrained lock.

Honestly, I don't think perf will improve (it will likely be slightly worse for ApplyBlocks as we'll have to take the lock a thousand times), but not having to call Done is a win safety-wise.