E2E tests: Add an initial validation stage

[mauromalara]

Description

The E2E tests have been developed in #2872 with the goal of being run manually. However, after such development, we have picked up a number of necessary changes that are common or impact all E2E tests.

This task aims to add an initial validation stage to E2E tests.

Initial validation stage

The initial validation stage will check via Ansible the following requirements on the managed nodes:

• Python version
• Linux distribution
• ...

[mauromalara]

Update 02/08/2022

I have researched Ansible roles, handlers, and others. Also, I have created a basic structure to implement the refactor.

Update 04/08/2022

I have changed the directory structure:

deps/wazuh_testing/wazuh_testing/end_to_end/
├── ansible_roles
│   ├── agents
│   │   ├── roles
│   │   │   └── main.yaml
│   │   └── vars
│   │       └── main.yaml
│   └── managers
│       ├── roles
│       │   └── main.yaml
│       └── vars
│           └── main.yaml
|   ...
└── __init__.py

Before a test, a group of tests, or all tests are executed, a fixture that validates the environment will be executed; it (which has session scope) uses the role path and the inventory path to execute the corresponding tasks.

The validation tasks will be executed in some/all hosts depending on the groups defined in the inventory:

• managers
• agents
• linux
• windows

All of the above options can be combined.

[mauromalara]

Update 05/08/2022

I tested the proposed solution from above, but it doesn't work as expected. So, I have been researched further and I tried this solution:

• The roles will be included in the configuration playbook itself, allowing the validation to run before
• The role path is defined inside the conftest.py
• The role will include a task/ folder and the main.yaml inside of it, this file contains all the validation tasks
• The role path is used in the configure_environment fixture, allowing

This way the pipeline that we will create in the future should only execute the test/s using pytest as a command and the inventory path as parameter.

I still have some doubts about the task itself, such as what would be the validations to be performed since, for example, in the role of Linux agent and manager tasks are shared, so I see the need to create a role more called "common".

[mauromalara]

Update 15/08/2022

Running the tests with the validation stage already developed I realized that:

1. All the current tests were only supposed to validate the environment and had no specific validation tasks.
2. With the previous design, it was necessary to launch 1 validation playbook for each selected test (even if the playbook repeated tasks).

This change allows to run a general validation prior to all tests (according to the tests that were selected to run) and then run (if any) the particular validation tasks for each test.

So, the validation phase is divided into 4 steps:

1. Step 1: Collect the data related to the selected tests that will be executed.
2. Step 2: Generate a playbook containing cross-checks for specified tests.
3. Step 3: Run the generated playbook.
4. Step 4: Execute test-specific validations (if any). It will run one validation for each selected test set.

The validation phase was executed beside the following tests:

• test_fim
• test_audit
• test_brute_force

Remaining tasks

1. Merge changes from the base branch into the working branch
2. Add a task to check the connection between Filebeat and the wazuh-indexer
3. Add a task to check the connection between Localhost and the wazuh-indexer
4. Add a task to check that all the Wazuh components are up and running.
5. Add a task in each test to validate the supported OS.

[mauromalara]

Update 16/08/2022

Done:

1. Merge changes from the base branch into the working branch
2. Add a task to check the connection between Filebeat and the wazuh-indexer
3. Add a task to check the connection between Localhost and the wazuh-indexer
4. Run a test to check if the validation phase works properly

Remain:

1. Add a task to check that all the Wazuh components are up and running. (This task will be cherry-picked from the @BelenValdivia development and included in the host_checker role)
2. Add a task in each test to validate the supported OS.

test_fim execution (6 test cases)