First pass implementation of the "Tool Meister"

[portante]

First pass implementation of the "Tool Meister"

See #1245.

The goal of the "Tool Meister" is to encapsulate the starting and stopping of tools into a wrapper daemon which is started once on each node for the duration of a benchmark script. Instead of the start/stop tools scripts using SSH to start/stop tools on local or remote hosts, a Redis Server is use to communicate with all the started Tool Meisters which execute the tool start/stop operations as a result of messages they receive using Redis's publish/subscribe pattern.

As a result of this encapsulation, we provide a separation of the orchestration of running tools remotely from how they are started and stopped. We have a CLI interface to start and stop the "Tool Meisters", which is a model that other orchestration implementations can follow (pbench-tool-meister-start, pbench-tool-meister-stop). We also have a CLI interface for communicating with the "Tool Meisters" to send the tool start & stop messages. However, this commit does not provide a Python3 API for that communication.

The Redis server location is passed as a set of parameters (host & port) to the Tool Meister instance, along with the name of a "key" in the Redis server which contains that Tool Meister's initial operating instructions for the duration of the benchmark script's execution:

• What Redis pub/sub channel to use
• What tool group describing the tools to use and their options

The Tool Meister then runs through a simple two phase life-cycle for tools until it is told to "terminate": "start" the registered tools on this host, and "stop" the registered tools on this host.

The initial expected phase is "start", where it waits to be told when to start its tools running from a published message on the "tool meister" channel. Once it starts one or more tools in the background via screen, it waits for a "stop" message to invoke the running tools' stop action.

This start/stop cycle is no different from the previous way tools were started and stopped, except that the start and stop operations no longer involve ssh operations to remote hosts.

Each start and stop message sent to the Tool Meisters is accompanied by two parameters: the tool group of the registered tool set (only used to ensure the context of the message is correct), and a path to a directory on the host (the controller) driving the benchmark where all the tool data will be collected.

Since the benchmark script ensures the directory is unique for each set of tool data collected (iteration / sample / host), the Tool Meister running on the same host as the controller just writes its collected tool data in that given directory.

However, when a Tool Meister is running on a host remote from the controller, that directory path is not present. Instead the remote Tool Meister uses a temporary directory instead of the given directory path. The given directory path is treated as a unique context ID to track all the tool data collected in temporary directories so that specific tool data can be retrieved when requested.

Because we are no longer using ssh to copy the collected tool data from the remote hosts to the local controller driving the benchmark, we have added a "send" phase for gathering each tool data set collected by a start / stop pair.

The controller running the benchmark driver determines when to request the collected tool data be "sent" back to a new Tool Data Sink process running on the controller. The send can be issued immediately following a stop, or all of the start/stop sequences can be executed before all the send requests are made, or some combination thereof. The only requirement is that a send has to follow its related start/stop sequence.

The Tool Data Sink is responsible for accepting data from remote Tool Meisters, via an HTTP PUT method, whenever a "send" message is posted.

Tool Meister Pseudo Code

The pseudo code for the use of the Tool Meisters in a benchmark script is as follows:

pbench-tool-meister-start  # New interface
for iter in ${iterations}; do
  for sample in ${samples}; do
    pbench-start-tools --group=${grp} --dir=${iter}/${sample}
    ... <benchmark> ...
    pbench-stop-tools --group=${grp} --dir=${iter}/${sample}
    # New interface added for `send` operation
    pbench-send-tools --group=${grp} --dir=${iter}/${sample}
    pbench-postprocess-tools --group=${grp} --dir=${iter}/${sample}
  done
done
pbench-tool-meister-stop  # New interface

Or having the tool data sent later:

pbench-tool-meister-start
for iter in ${iterations}; do
  for sample in ${samples}; do
    pbench-start-tools --group=${grp} --dir=${iter}/${sample}
    ... <benchmark> ...
    pbench-stop-tools --group=${grp} --dir=${iter}/${sample}
  done
done
for iter in ${iterations}; do
  for sample in ${samples}; do
    pbench-send-tools --group=${grp} --dir=${iter}/${sample}
    pbench-postprocess-tools --group=${grp} --dir=${iter}/${sample}
  done
done
pbench-tool-meister-stop

Note the addition of the new pbench-send-tools interface a caller can use to indicate when remote tool data can be sent.

Tool Post-Processing Locally

A behavioral change that comes with this work is that tool post-processing is no longer performed remotely on the host where it is collected. Previous work added the necessary "stop-post-processing" step, so that when tools are stopped any necessary post-processing and environmental data collection required to allow the tool data to be used off-host is collected.

Removal of Remote Tool Registration

This work IMPLIES that we no longer need to record registered tools remotely. We only need to start a Tool Meister remotely for each host, passing the initial data it needs at start time via Redis.

Now that pbench-register-tool[-set] supports the ability for a caller to register a tool [or tool set] for a list of hosts, we keep all the tool data local on the pbench "controller" node where the pbench-agent's user registers tools.

By doing this, we remove the need to manage a distributed data set across multiple hosts, allowing for a "late" binding of tools to be run on a set of hosts. In other words, the tool registration can be done without a host being present, with the understanding that it must be present when a workload is run.

This is particularly powerful for environments like, OpenStack and OpenShift, where software installation of tools are provided by container images, VM images (like qcow2), and other automated installation environments.

This is an invasive change, as knowledge about how tool data is represented on disk was spread out across different pieces of code. We have attempted to consolidate that knowledge, future work might be required to adhere to the DRY principle.

NOTES:

• The Tool Meister invokes the existing tools in tool-scripts as they operate today without any changes

Work not addressed by this commit

• Rewrite pbench-tool-trigger into a python application that talks directly to the Redis server to initiate the start, stop, send messages

• Add support for the Tool Meisters to support collecting the pbench-sysinfo-dump data

• Provide Python3 APIs for communicating with the Tool Meisters

[portante]

This works now depends on PR #1317.

[dbutenhof]

Let it go.

[zulcss]

+1

[portante]

We have two approvals to merge this, @Maxusmusti, do you want to add yours officially?

@ndokos, @npalaska, @riya-17, do you want us to wait for your reviews before we merge?

[Maxusmusti]

We have two approvals to merge this, @Maxusmusti, do you want to add yours officially?

@ndokos, @npalaska, @riya-17, do you want us to wait for your reviews before we merge?

I approve as well!

[npalaska]

I did some more tests on this branch and everything seems to be fine. Sure I might have missed few bugs (if there are any) but it can be fixed incrementally with additional PRs