Swap performance testing implementation to use Hyperfoil.

[whitingjr]

This PR will replace the JMeter injector with Hyperfoil.
Included in this PR are an integration with Ansible playbooks. Updates to the documentation for users to get started with the replacement injector.

[eguzki]

Some changes are requested with initial review.

The initial review relate to an outdated commit.

[eguzki]

In README.md, TOC has not been updated

[eguzki]

what is deployment/files/report.sh file for? It is too big and git does not show and my IDE has issues opening it.

It contains

#TODO create script :)

:O

[whitingjr]

The purpose of deployment/files/report.sh is for generating the report html file from the JSON data file.
On second thoughts I will remove this file. Replacing it with a containerized version. To generate the report with a docker run command.

[eguzki]

some comments left. Waiting for feedback

[whitingjr]

Feedback provided to comments.

[davidor]

I'm not familiar with the codebase, so I will not comment about the code changes, but I'd like to give my opinion as a user of the tool.

We've been using this tool to benchmark many versions of 3scale. Jmeter provides all we need, which includes things like: ability to configure the number of connections, the throughput, and nice reports. Why are we changing to Hyperfoil? What problems does it solve?

Also, I'm a bit concerned about changing the tool because from the moment we change, we won't be able to compare with previous results. Has anyone tried to run the same benchmarks we've run for the latest version of 3scale with hyperfoil to see how results compare?

[whitingjr]

We've been using this tool to benchmark many versions of 3scale. Jmeter provides all we need, which includes things like: ability to configure the number of connections, the throughput, and nice reports. Why are we changing to Hyperfoil? What problems does it solve?

Also, I'm a bit concerned about changing the tool because from the moment we change, we won't be able to compare with previous results. Has anyone tried to run the same benchmarks we've run for the latest version of 3scale with hyperfoil to see how results compare?

Swapping out the driver to use Hyperfoil has been advocated so that load testing can be achieved easily in a microservices architecture. The driver also supports the ability to distribute the load generation. Amongst one or more Agents.
You can still configure number of connections, throughput and get a nice report afterwards.
Doing a baseline run comparison will give you a good idea of comparability. I'll send you a report (offline) I created that demonstrates the measured difference for a prior (0.6) version of Hyperfoil.

[eguzki]

JMeter also supports Distributed Testing https://jmeter.apache.org/usermanual/jmeter_distributed_testing_step_by_step.html

[eguzki]

The following params

shared_connections: 64
injector_hyperfoil_target_port: 443

are duplicated in several files called

deployment/group_vars/all.yml

and

deployment/inventory/group_vars/all.yml

Please, make sure more attributes are not duplicated and only used files are being committed.

[whitingjr]

Maybe, we should remove "CONCURRENCY" from the exposed configuration. And have shared_connections a value big enough to have all agents/executors get at least 1 connection.

I was thinking the same.

[eguzki]

I just tested this branch, and still get the same error:

TASK [include_role : hyperfoil.hyperfoil_test] ***********************************************************************************************************************************************************************************************************************************
task path: /home/eguzki/git/perftest-toolkit2/deployment/run.yml:4
ERROR! conflicting action statements: shell, warn

The error appears to be in '/home/eguzki/.ansible/roles/hyperfoil.hyperfoil_test/tasks/main.yml': line 34, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

    test_files: []
- name: Upload benchmark template
  ^ here

[whitingjr]

I just tested this branch, and still get the same error:

TASK [include_role : hyperfoil.hyperfoil_test] ***********************************************************************************************************************************************************************************************************************************
task path: /home/eguzki/git/perftest-toolkit2/deployment/run.yml:4
ERROR! conflicting action statements: shell, warn

The error appears to be in '/home/eguzki/.ansible/roles/hyperfoil.hyperfoil_test/tasks/main.yml': line 34, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

    test_files: []
- name: Upload benchmark template
  ^ here

Can you raise an issue on the hyperfoil_test project here please ?

[eguzki]

It would be better if you do to explain configuration and environment. I am not familiar with hyperfoil parameters and configuration.

[whitingjr]

It would be better if you do to explain configuration and environment. I am not familiar with hyperfoil parameters and configuration.

It isn't anything to do with Hyperfoil as far as I can see. Don't you think this is an Ansible error reported for the shell module execution ?
Have you tried either:
a) editing /home/eguzki/.ansible/roles/hyperfoil.hyperfoil_test/tasks/main.yml to remove the warn parameter. On line 37
b) adding an indent for the warn parameter.

- name: Upload benchmark template 
  shell:
      "{{ lookup('template', 'templates/curl.j2') }} \
    http://{{ hyperfoil_controller_host }}:{{ hyperfoil_controller_port }}/benchmark"
  register: curl_cmd
  failed_when: curl_cmd.rc > 1
    warn: no

then execute the run.yml playbook again to see if this solves the problem ?

[eguzki]

If you suggest to modify some third party source of code, please open issue to that repo suggesting to them, not to me. If it fails to me, it could fail to anyone else using it.

[whitingjr]

If you suggest to modify some third party source of code, please open issue to that repo suggesting to them, not to me. If it fails to me, it could fail to anyone else using it.

I don't mind creating the issue against the hyperfoil_test project for you to help you with this issue.
Can you at least attempt trying the fix b) I've suggested above to the file /home/eguzki/.ansible/roles/hyperfoil.hyperfoil_test/tasks/main.yml. Then communicate with me with the results of this fix. To see if that solves the problem on your environment ?

[eguzki]

Even worse message, Syntax Error.

TASK [include_role : hyperfoil.hyperfoil_test] ***********************************************************************************************************************************************************************************************************************************
task path: /home/eguzki/git/perftest-toolkit2/deployment/run.yml:4
ERROR! Syntax Error while loading YAML.
  mapping values are not allowed in this context

The error appears to be in '/home/eguzki/.ansible/roles/hyperfoil.hyperfoil_test/tasks/main.yml': line 40, column 9, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

  failed_when: curl_cmd.rc > 1
    warn: no
        ^ her

Does it work for you?

[whitingjr]

Does it work for you?

On a RHEL/Fedora platform it has always worked.
Thanks for attempting by adding an indent. Can you do another attempt please. This time removing the offending line ?

[eguzki]

Last attempt. This PR is not the place to test your roles.

fatal: [192.168.57.3]: FAILED! => {"changed": false, "msg": "AnsibleUndefinedVariable: 'dict object' has no attribute 'injector_target'"}

[whitingjr]

Last attempt. This PR is not the place to test your roles.

fatal: [192.168.57.3]: FAILED! => {"changed": false, "msg": "AnsibleUndefinedVariable: 'dict object' has no attribute 'injector_target'"}

From this test it looks like the issue with the Ansible "conflicting action statements: shell, warn" has been solved. I'll raise an issue with the relevant hyperfoil_test project.
Created issue is here.

[eguzki]

Is it ready to be tested again?

I see there are new changes. New review is needed.

[whitingjr]

Is it ready to be tested again?

I see there are new changes. New review is needed.

Yes it is ready to be tested.

[eguzki]

Proposed refactor: #54

waiting for you input @whitingjr

[eguzki]

Another proposed refactor: #59