Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include markers for each system test environment #3961

Merged
merged 7 commits into from
Feb 17, 2023

Conversation

Rebits
Copy link
Member

@Rebits Rebits commented Feb 16, 2023

Related issue
#3960

Description

This PR includes markers for each system test environment.

Added

  • Markers for each system test environment

Testing performed

Tester Test path Jenkins Local OS Commit Notes
@user (Developer) 🚫 🚫 🚫 🚫 🚫 🚫 Nothing to highlight
@user (Reviewer) 🚫 🚫 🚫 🚫 🚫 🚫 Nothing to highlight

@Rebits Rebits self-assigned this Feb 16, 2023
@Rebits Rebits force-pushed the 3960-include-marks-for-system-environments branch from 2fa13bb to dfc711c Compare February 16, 2023 18:31
@Rebits
Copy link
Member Author

Rebits commented Feb 17, 2023

Testing

enrollment_cluster

$ python -m pytest . --collect-only -m enrollment_cluster_env
================================================ test session starts ================================================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-0.13.1
rootdir: /home/rebits/Wazuh/wazuh-qa/3960-include-marks-for-system-environments/tests/system, configfile: pytest.ini
plugins: html-3.1.1, testinfra-5.0.0, json-report-1.5.0, metadata-2.0.1, docgen-1.3.0
collected 121 items / 103 deselected / 18 selected                                                                  

<Module test_cluster/test_agent_enrollment/test_agent_enrollment.py>
  <Function test_agent_enrollment>
<Package test_agent_groups>
  <Module test_agent_default_group_added.py>
    <Function test_agent_default_group_added[wazuh-master]>
    <Function test_agent_default_group_added[wazuh-worker1]>
    <Function test_agent_default_group_added[wazuh-worker2]>
  <Module test_agent_groups.py>
    <Function test_agent_groups_create_remove_group>
  <Module test_assign_agent_group_with_enrollment.py>
    <Function test_assign_agent_to_a_group[wazuh-worker1]>
  <Module test_assign_agent_never_connected_to_group.py>
    <Function test_assign_agent_to_a_group[wazuh-master]>
    <Function test_assign_agent_to_a_group[wazuh-worker1]>
  <Module test_assign_agent_to_a_group_api.py>
    <Function test_assign_agent_to_a_group[wazuh-master-active]>
    <Function test_assign_agent_to_a_group[wazuh-master-disconnected]>
    <Function test_assign_agent_to_a_group[wazuh-worker1-active]>
    <Function test_assign_agent_to_a_group[wazuh-worker1-disconnected]>
  <Module test_assign_agent_to_a_group_by_tool.py>
    <Function test_assign_agent_to_a_group_by_tool[wazuh-master]>
    <Function test_assign_agent_to_a_group_by_tool[wazuh-worker1]>
  <Module test_assign_groups_guess.py>
    <Function test_assign_agent_to_a_group[wazuh-master-0]>
    <Function test_assign_agent_to_a_group[wazuh-master-1]>
    <Function test_assign_agent_to_a_group[wazuh-worker1-0]>
    <Function test_assign_agent_to_a_group[wazuh-worker1-1]>

basic_cluster
$ python -m pytest. --collect-only -m basic_cluster_env
============================================================================================================ test session starts =============================================================================================================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-0.13.1
rootdir: /home/rebits/Wazuh/wazuh-qa/3960-include-marks-for-system-environments/tests/system, configfile: pytest.ini
plugins: html-3.1.1, testinfra-5.0.0, json-report-1.5.0, metadata-2.0.1, docgen-1.3.0
collected 121 items / 110 deselected / 11 selected                                                                                                                                                                                           

<Module test_cluster/test_agent_files_deletion/test_agent_files_deletion.py>
  <Function test_agent_files_deletion>
<Package test_agent_groups>
  <Module test_agent_groups_forced_change.py>
    <Function test_sync_when_forced_to_change_a_group[wazuh-master]>
    <Function test_sync_when_forced_to_change_a_group[wazuh-worker1]>
    <Function test_force_group_change_during_sync>
<Module test_cluster/test_agent_info_sync/test_agent_info_sync.py>
  <Function test_agent_info_sync>
  <Function test_agent_info_sync_remove_agent>
<Module test_cluster/test_agent_key_polling/test_agent_key_polling.py>
  <Function test_agent_key_polling>
<Module test_multigroups/test_multigroups.py>
  <Function test_multigroups_not_reloaded>
  <Function test_multigroups_updated[test_mg_0]>
  <Function test_multigroups_updated[default]>
  <Function test_multigroups_deleted>
big_cluster_40_agents
$ python -m pytest. --collect-only -m big_cluster_40_agents_env
<Package test_agent_groups>
  <Module test_groups_sync_default.py>
    <Function test_agent_groups_sync_default[wazuh-master]>
    <Function test_agent_groups_sync_default[wazuh-worker1]>
  <Module test_groups_sync_time.py>
    <Function test_agent_groups_sync_time[wazuh-master]>
    <Function test_agent_groups_sync_time[wazuh-worker1]>
four_manager_disconnected_node
<Package test_agent_groups>
  <Module test_agent_groups_new_cluster_node.py>
    <Function test_agent_groups_sync_when_add_a_new_cluster_node>
    <Function test_agent_groups_sync_worker_new_node>
agentless_cluster
$ python -m pytest . --collect-only -m agentless_cluster_env
============================================================================================================ test session starts =============================================================================================================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-0.13.1
rootdir: /home/rebits/Wazuh/wazuh-qa/3960-include-marks-for-system-environments/tests/system, configfile: pytest.ini
plugins: html-3.1.1, testinfra-5.0.0, json-report-1.5.0, metadata-2.0.1, docgen-1.3.0
collected 121 items / 97 deselected / 24 selected                                                                                                                                                                                            

<Module test_cluster/test_integrity_sync/test_integrity_sync.py>
  <Function test_missing_file>
  <Function test_shared_files>
  <Function test_extra_files>
  <Function test_zip_size_limit>
<Module test_cluster/test_ruleset_sync_status/test_ruleset_sync_status.py>
  <Function test_ruleset_sync_status>
<Module test_jwt_invalidation/test_change_rbac_mode.py>
  <Function test_change_rbac_mode_with_endpoint[login_endpoint0]>
  <Function test_change_rbac_mode_with_endpoint[login_endpoint1]>
  <Function test_change_rbac_mode_manually[login_endpoint0]>
  <Function test_change_rbac_mode_manually[login_endpoint1]>
<Module test_jwt_invalidation/test_change_security_resources.py>
  <Function test_change_user_policy_and_role[wazuh-master-wazuh-worker1-login_endpoint0]>
  <Function test_change_user_policy_and_role[wazuh-master-wazuh-worker1-login_endpoint1]>
  <Function test_change_user_policy_and_role[wazuh-worker1-wazuh-worker2-login_endpoint0]>
  <Function test_change_user_policy_and_role[wazuh-worker1-wazuh-worker2-login_endpoint1]>
<Module test_jwt_invalidation/test_disconnected_nodes.py>
  <Function test_create_user_when_node_is_disconnected>
<Module test_jwt_invalidation/test_revoke_endpoint.py>
  <Function test_revoke_all_tokens
one_manager_agent
python -m pytest . --collect-only -m one_manager_agent_env
============================================================================================================ test session starts =============================================================================================================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-0.13.1
rootdir: /home/rebits/Wazuh/wazuh-qa/3960-include-marks-for-system-environments/tests/system, configfile: pytest.ini
plugins: html-3.1.1, testinfra-5.0.0, json-report-1.5.0, metadata-2.0.1, docgen-1.3.0
collected 121 items / 115 deselected / 6 selected                                                                                                                                                                                            

<Package test_synchronization>
  <Module test_synchronization.py>
    <Function test_synchronization[testdir1-add-wazuh-agent1]>
    <Function test_synchronization[testdir1-add-wazuh-manager]>
    <Function test_synchronization[testdir1-modify-wazuh-agent1]>
    <Function test_synchronization[testdir1-modify-wazuh-manager]>
    <Function test_synchronization[testdir1-delete-wazuh-agent1]>
    <Function test_synchronization[testdir1-delete-wazuh-manager]>
manager_agent
<Module test_active_response_log_format/test_active_response_log_format.py>
  <Function test_active_response_log_format[wazuh-agent1-data/messages_415_or_lower.yml]>
  <Function test_active_response_log_format[wazuh-agent2-data/messages_420_to_424.yml]>
  <Function test_active_response_log_format[wazuh-agent3-data/messages_425_or_greater.yml]>

@Rebits
Copy link
Member Author

Rebits commented Feb 17, 2023

Detected unexpected behaviour in markers logic

  • The package wazuh_testing __init__ file define some markers improperly, producing the following warnings:
============================================================================================================== warnings summary ==============================================================================================================
../../../../../.local/lib/python3.10/site-packages/wazuh_testing-4.4.0-py3.10.egg/wazuh_testing/modules/__init__.py:27
  /home/rebits/.local/lib/python3.10/site-packages/wazuh_testing-4.4.0-py3.10.egg/wazuh_testing/modules/__init__.py:27: PytestUnknownMarkWarning: Unknown pytest.mark.win32 - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    WINDOWS = pytest.mark.win32

../../../../../.local/lib/python3.10/site-packages/wazuh_testing-4.4.0-py3.10.egg/wazuh_testing/modules/__init__.py:28
  /home/rebits/.local/lib/python3.10/site-packages/wazuh_testing-4.4.0-py3.10.egg/wazuh_testing/modules/__init__.py:28: PytestUnknownMarkWarning: Unknown pytest.mark.linux - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    LINUX = pytest.mark.linux

../../../../../.local/lib/python3.10/site-packages/wazuh_testing-4.4.0-py3.10.egg/wazuh_testing/modules/__init__.py:29
  /home/rebits/.local/lib/python3.10/site-packages/wazuh_testing-4.4.0-py3.10.egg/wazuh_testing/modules/__init__.py:29: PytestUnknownMarkWarning: Unknown pytest.mark.darwin - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    MACOS = pytest.mark.darwin

../../../../../.local/lib/python3.10/site-packages/wazuh_testing-4.4.0-py3.10.egg/wazuh_testing/modules/__init__.py:30
  /home/rebits/.local/lib/python3.10/site-packages/wazuh_testing-4.4.0-py3.10.egg/wazuh_testing/modules/__init__.py:30: PytestUnknownMarkWarning: Unknown pytest.mark.sunos5 - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    SOLARIS = pytest.mark.sunos5

../../../../../.local/lib/python3.10/site-packages/wazuh_testing-4.4.0-py3.10.egg/wazuh_testing/modules/__init__.py:32
  /home/rebits/.local/lib/python3.10/site-packages/wazuh_testing-4.4.0-py3.10.egg/wazuh_testing/modules/__init__.py:32: PytestUnknownMarkWarning: Unknown pytest.mark.agent - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    AGENT = pytest.mark.agent

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
  • ValueError for integration tests collection
$ python -m pytest . -m linux
../../../.local/lib/python3.10/site-packages/_pytest/config/__init__.py:1488: in getoption
    val = getattr(self.option, name)
E   AttributeError: 'Namespace' object has no attribute 'fim_database_memory'

The above exception was the direct cause of the following exception:
../../../.local/lib/python3.10/site-packages/pluggy/manager.py:127: in register
    hook._maybe_apply_history(hookimpl)
../../../.local/lib/python3.10/site-packages/pluggy/hooks.py:333: in _maybe_apply_history
    res = self._hookexec(self, [method], kwargs)
../../../.local/lib/python3.10/site-packages/pluggy/manager.py:93: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
../../../.local/lib/python3.10/site-packages/pluggy/manager.py:84: in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
tests/integration/conftest.py:325: in pytest_configure
    fim_database_memory = config.getoption("--fim-database-memory")
../../../.local/lib/python3.10/site-packages/_pytest/config/__init__.py:1499: in getoption
    raise ValueError(f"no option named {name!r}") from e
E   ValueError: no option named 'fim_database_memory'

@jmv74211 jmv74211 merged commit e7be294 into 4.4 Feb 17, 2023
@jmv74211 jmv74211 deleted the 3960-include-marks-for-system-environments branch February 17, 2023 11:59
juliamagan added a commit that referenced this pull request Feb 20, 2023
Deblintrake09 pushed a commit that referenced this pull request Apr 24, 2023
* feat(#3960): include markers for system env

* docs(#3960): include #3961 changelog

* fix(#3960): group sync default marker

* style(#3960): format markers

* style(#3960): pycodestyle errors

* fix(#3960): test_integrity_sync env marker
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants