An actorless load generator

[danielmitterdorfer]

At the moment Rally uses a process per client model (see #9 (comment), #58 and fe5dba3 for more details) and implements this with an actor system that allows clients (actors) to communicate with each other. The basic structure is that there is a controller actor (called Driver) for all clients and each client (actor) is responsible to execute scheduled tasks. However, this has several drawbacks that we've seen in practice:

• Complexity added by the actor framework: we need to spin up a separate process to communicate across nodes. Having the ability to communicate across nodes was mostly needed to spin up a cluster from the coordinator machine (where the benchmark has been started) but is less important for the load generator (although we also use it to distribute it across several machines). This was convenient in the early days but nowadays we let other tools handle remote connections during setup (e.g. Ansible) and use Rally only to setup the individual nodes. Thus we don't need an actor system as much as we did before.
• Terminal output issues (the actor system performs a double-fork and thus we cannot write to the terminal)
• Issues to even startup the system on Windows that are hard to debug.
• Limit on the upper bound of supported clients (the whole system is just hanging if there are too many actors).
• Logging issues: logs are sent to a central component which then writes to the log file. If one actor issues too many logs, the actor system just becomes unresponsive (which is quite hard to debug because there are no logs to begin with).
• Debugging actor issues is quite tricky and we don't have much influence over what is logged.

Note that these limitations are due to the specific actor system implementation, and not necessarily conceptual issues. There are, however, also shortcomings that can be related to the process per client model:

• There is no built-in mechanism for clients to share state (this is not technically impossible, it's just a bit harder than with a single process model). Sharing state can be sometimes convenient e.g. if in an initial step, data is retrieved from Elasticsearch and then we can use this information in later steps. This is done for example by the eventdata track that retrieves the minimum and maximum date of log entries in Elasticsearch and uses it to choose appropriate date range parameters for simulating dashboard queries in later steps.
• Starting a process for each client is quite heavy-weight and thus it's not feasible to simulate thousands of clients with Rally.

We should explore a few options based on these shortcomings:

• Using the Python Elasticsearch async client.
• Using asyncio directly
• Keeping the process per client model but instead use multiprocessing

There are maybe other options as well but these are the choices that first come to mind. We should also refine this ticket as we proceed with our analysis.

IMHO it is ok to support only one load generator machine in the first step but we should aim to support the Rally tracks that we currently maintain.

[danielmitterdorfer]

Reopening as our current multi-core implementation in #944 still uses the actor system.

[danielmitterdorfer]

Closing as we have no intention of changing this in the foreseeable future (this may change in the far future though).