Investigate and document tests - System E2E tests

[Deblintrake09]

Parent issue	#4241
Pipeline state	🟡 being fixed in https://github.com/wazuh/wazuh-jenkins/issues/5237
Tests state	🟡 generally working - some skipped modules
Jenkins link	https://ci.wazuh.info/job/Test_e2e_system
Branch used	4.4

Description

This issue aims to analyze and document all about the Systems E2E test. Jenkins parameters, pytest location, steps in the test and current state of it.

State

• The test suite has tests in different states. As described in the comment.
• The pipeline currently has dependency issues so tests have been executed with branch 5237-fix-test_e2e_system-pipeline-due-to-dependency-upgrade-error
• During release a series of tests are done Manually and are not currently automated. Last tests to have been executed were done in Release 4.4.2-RC1 and are tracked in Release 4.4.2 - RC 1 - E2E UX tests wazuh#16965

Parameters

• Base general parameters
  • VERBOSITY: verbosity level for logs
  • JENKINS_REFERENCE: wazuh-jenkins repository version
  • ANSIBLE_REFERENCE: Ansible repo version
  • DESTROY_INSTANCES: If checked, instances are destroyed immediately after tests finish
  • MAXIMUM_UPTIME: maximum time Instances will be up.
  • CHECKRUN_REFERENCE
  • DEBUG: enables debug mode.
  • USE_ACTIVE_ENVIRONMENT: If enabled can used Invetory passed in ENVIRONMENT_INVENTORY field for testing
  • ENVIRONMENT_INVENTORY: Enviroment inventory data.
• Base parameters for all tests:
  • TARGET_PKG_VERSION: Package version to install
  • TARGET_PKG_REVISION: package revision
  • TARGET_REPOSITORY: repository where package is located
• Base parameters for QA tests:
  -QA_REFERENCE: wazuh-qa repository branch
• Specific parameters:
  • TESTS: path of tests to be executed
  • TIERS: test tiers to execute. Launch all test tiers by default

[Deblintrake09]

Test Executions

Module	Result	Notes
test_audit	🟢
test_aws_infrastructure_monitoring	⚫	Skipped because of Issue #3211
test_brute_force	🟢
test_detect_suspicious_binaries	🟢
test_docker	⚫	Skipped because of Issue #3210
test_emotet	🟢
test_fim	⚫	Skipped because of Issue #3207
test_ip_reputation	🟢
test_osquery_integration	🟢
test_shellshock_attack_detection	🔴 🔴	No associated issue to fails. Requires more research
test_slack_integration	⚫	Skipped because of Issue #3286
test_sql_injection	🟢
test_suricata_integration	🟢
test_unauthorized_processes_detection	🟢
test_virustotal_integration	🟢
test_vulnerability_detector	🟡	Test fixes are being done in #3219
test_windows_defender	🟢
test_yara_integration	🟢

Status	Description
⚫	Not executed / Skipped in code - Check notes
🔵	In execution
🟢	Passed / Works properly
🟡	Skipped - Being actively fixed - Check notes
🔴	Tests failed

[Deblintrake09]

Manual E2E testing

• During release a series of tests are done Manually and are not currently automated. Last tests to have been executed were done in Release 4.4.2-RC1 and are tracked in Release 4.4.2 - RC 1 - E2E UX tests wazuh#16965

[verdx]

LGTM

[QU3B1M]

LGTM!!

[juliamagan]

We could add that the pipeline not only launches the tests, but also uses WazuhQAEnvironment for the necessary provisioning of the tests.

Currently, it only launches the E2E tests, but in the future, this pipeline should be able to launch the system tests as well.

Finally, the pipeline parameters are shown, but not explained.

The link to the spreadsheet with the documentation is in the parent issue.

[davidjiglesias]

LGTM!