AWS integration tests #3911

nico-stefani · 2023-02-08T13:19:51Z

Related issue
#3333

Description

Added

CloudTail integration tests
VPC integration tests
Config integration tests
ALB, CLB, and NLB integration tests
KMS integration tests
Macie integration tests
Trusted advisor integration tests
GuardDuty integration tests
WAF integration tests
Server Access integration tests
Inspector integration tests
CloudWatch integration tests
Umbrella integration tests
Parser integration tests

Testing performed

Service	Tier 0	Tier 1	Related Issues
CloudTrail	🟢	🟢
VPC	🟢	🟢	wazuh/wazuh#16138 wazuh/wazuh#15763
Config	🟢	🟢	wazuh/wazuh#16138
ALB	🟡	🟢	wazuh/wazuh#15804
CLB	🟡	🟢	wazuh/wazuh#15804
NLB	🟡	🟢	wazuh/wazuh#15804
KMS	🟢	🟢
Macie	🟢	🟢
Trusted Advisor	🟢	🟢
GuardDuty	🟢	🟢	wazuh/wazuh#16246
WAF	🟢	🟢	wazuh/wazuh#16290 wazuh/wazuh#16246
Server Access	🟢	🟢	wazuh/wazuh#16290 wazuh/wazuh#16246
Inspector	🟢	🟡	wazuh/wazuh#16301 wazuh/wazuh#16303
CloudWatch	🟢	🟢	wazuh/wazuh#16301 wazuh/wazuh#16375
Umbrella	🟢	🟢	wazuh/wazuh#16246
Parser	🟢	🟢	wazuh/wazuh#16383

Tier 0

root@ip-172-31-82-0:/home/ubuntu/wazuh-qa/tests/integration/test_aws# pytest --disable-warnings --tb=line --tier 0
================================================================================================ test session starts ================================================================================================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/ubuntu/wazuh-qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.4, html-3.1.1, testinfra-5.0.0
collected 195 items / 16 deselected / 179 selected

test_basic.py ................                                                                                                                                                                                [  8%]
test_discard_regex.py ..............                                                                                                                                                                          [ 16%]
test_log_groups.py ..                                                                                                                                                                                         [ 17%]
test_only_logs_after.py ...............................                                                                                                                                                       [ 35%]
test_parser.py ..........................                                                                                                                                                                     [ 49%]
test_path.py ..........................................                                                                                                                                                       [ 73%]
test_path_suffix.py .........                                                                                                                                                                                 [ 78%]
test_regions.py ........................                                                                                                                                                                      [ 91%]
test_remove_from_bucket.py ...sss.........                                                                                                                                                                    [100%]

============================================================================ 176 passed, 3 skipped, 16 deselected in 3645.39s (1:00:45) =============================================================================

Tier 1

root@ip-172-31-82-0:/home/ubuntu/wazuh-qa/tests/integration/test_aws# pytest --disable-warnings --tb=line --tier 1
================================================================================================ test session starts ================================================================================================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/ubuntu/wazuh-qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.4, html-3.1.1, testinfra-5.0.0
collected 195 items / 179 deselected / 16 selected

test_only_logs_after.py ..............x.                                                                                                                                                                      [100%]

============================================================================= 15 passed, 179 deselected, 1 xfailed in 362.17s (0:06:02) =============================================================================

nico-stefani · 2023-02-08T21:24:57Z

CloudTrail Results

Tier 0

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 99 items / 85 deselected / 14 selected

test_basic.py .                                                          [  7%]
test_discard_regex.py .                                                  [ 14%]
test_only_logs_after.py ..                                               [ 28%]
test_path.py ...                                                         [ 50%]
test_path_suffix.py ...                                                  [ 71%]
test_regions.py ...                                                      [ 92%]
test_remove_from_bucket.py .                                             [100%]

=============================== warnings summary ===============================
../../../../../../usr/local/lib/python3.10/dist-packages/_pytest/cacheprovider.py:433
  /usr/local/lib/python3.10/dist-packages/_pytest/cacheprovider.py:433: PytestCacheWarning: cache could not write path /home/vagrant/qa/tests/integration/.pytest_cache/v/cache/nodeids
    config.cache.set("cache/nodeids", sorted(self.cached_nodeids))

../../../../../../usr/local/lib/python3.10/dist-packages/_pytest/stepwise.py:52
  /usr/local/lib/python3.10/dist-packages/_pytest/stepwise.py:52: PytestCacheWarning: cache could not write path /home/vagrant/qa/tests/integration/.pytest_cache/v/cache/stepwise
    session.config.cache.set(STEPWISE_CACHE_DIR, [])

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========== 14 passed, 85 deselected, 2 warnings in 309.55s (0:05:09) ===========

Tier 1

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 81 items / 80 deselected / 1 selected

test_only_logs_after.py .                                                [100%]

=============================== warnings summary ===============================
================ 1 passed, 80 deselected, 2 warnings in 23.78s =================

nico-stefani · 2023-02-09T19:40:05Z

VPC Results

Tier 0

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 99 items / 85 deselected / 14 selected

test_basic.py .                                                          [  7%]
test_discard_regex.py F                                                  [ 14%]
test_only_logs_after.py .F                                               [ 28%]
test_path.py F..                                                         [ 50%]
test_path_suffix.py F..                                                  [ 71%]
test_regions.py FFF                                                      [ 92%]
test_remove_from_bucket.py .                                             [100%]

=================================== FAILURES ===================================
____________________ test_discard_regex[vpc_discard_regex] _____________________
=========================== short test summary info ============================
FAILED test_discard_regex.py::test_discard_regex[vpc_discard_regex] - Timeout...
FAILED test_only_logs_after.py::test_with_only_logs_after[vpc_with_only_logs_after]
FAILED test_path.py::test_path[vpc_path_with_data] - TimeoutError: The AWS mo...
FAILED test_path_suffix.py::test_path_suffix[vpc_path_suffix_with_data] - Tim...
FAILED test_regions.py::test_regions[vpc_region_with_data] - TimeoutError: Th...
FAILED test_regions.py::test_regions[vpc_regions_with_data] - TimeoutError: T...
FAILED test_regions.py::test_regions[vpc_inexistent_region] - TimeoutError: T...
====== 7 failed, 7 passed, 85 deselected, 3 warnings in 433.79s (0:07:13) ======

Tier 1

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 99 items / 98 deselected / 1 selected

test_only_logs_after.py F                                                [100%]

=================================== FAILURES ===================================
=========================== short test summary info ============================
FAILED test_only_logs_after.py::test_multiple_calls[vpc_only_logs_after_multiple_calls]
=========== 1 failed, 98 deselected, 3 warnings in 69.97s (0:01:09) ============

nico-stefani · 2023-02-09T19:51:41Z

Config Results

Tier 0

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 99 items / 85 deselected / 14 selected

test_basic.py .                                                          [  7%]
test_discard_regex.py F                                                  [ 14%]
test_only_logs_after.py .F                                               [ 28%]
test_path.py F..                                                         [ 50%]
test_path_suffix.py ...                                                  [ 71%]
test_regions.py FFF                                                      [ 92%]
test_remove_from_bucket.py .                                             [100%]

=================================== FAILURES ===================================
___________________ test_discard_regex[config_discard_regex] ___________________
=========================== short test summary info ============================
FAILED test_discard_regex.py::test_discard_regex[config_discard_regex] - Time...
FAILED test_only_logs_after.py::test_with_only_logs_after[config_with_only_logs_after]
FAILED test_path.py::test_path[config_path_with_data] - TimeoutError: The AWS...
FAILED test_regions.py::test_regions[config_region_with_data] - TimeoutError:...
FAILED test_regions.py::test_regions[config_regions_with_data] - TimeoutError...
FAILED test_regions.py::test_regions[config_inexistent_region] - TimeoutError...
====== 6 failed, 8 passed, 85 deselected, 3 warnings in 431.28s (0:07:11) ======

Tier 1

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 99 items / 98 deselected / 1 selected

test_only_logs_after.py F                                                [100%]

=================================== FAILURES ===================================
FAILED test_only_logs_after.py::test_multiple_calls[config_only_logs_after_multiple_calls]
=========== 1 failed, 98 deselected, 3 warnings in 60.59s (0:01:00) ============

nico-stefani · 2023-02-09T20:00:18Z

ALB Results

Tier 0

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 99 items / 88 deselected / 11 selected

test_basic.py .                                                          [  9%]
test_discard_regex.py .                                                  [ 18%]
test_only_logs_after.py ..                                               [ 36%]
test_path.py ...                                                         [ 63%]
test_regions.py ...                                                      [ 90%]
test_remove_from_bucket.py s                                             [100%]

===== 10 passed, 1 skipped, 88 deselected, 2 warnings in 212.11s (0:03:32) =====

Tier 1

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 99 items / 98 deselected / 1 selected

test_only_logs_after.py .                                                [100%]

================ 1 passed, 98 deselected, 2 warnings in 35.40s =================

nico-stefani · 2023-02-09T20:09:04Z

CLB Results

Tier 0

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 99 items / 88 deselected / 11 selected

test_basic.py .                                                          [  9%]
test_discard_regex.py .                                                  [ 18%]
test_only_logs_after.py ..                                               [ 36%]
test_path.py ...                                                         [ 63%]
test_regions.py ...                                                      [ 90%]
test_remove_from_bucket.py s                                             [100%]

=============================== warnings summary ===============================
../../../../../../usr/local/lib/python3.10/dist-packages/_pytest/cacheprovider.py:433
  /usr/local/lib/python3.10/dist-packages/_pytest/cacheprovider.py:433: PytestCacheWarning: cache could not write path /home/vagrant/qa/tests/integration/.pytest_cache/v/cache/nodeids
    config.cache.set("cache/nodeids", sorted(self.cached_nodeids))

../../../../../../usr/local/lib/python3.10/dist-packages/_pytest/stepwise.py:52
  /usr/local/lib/python3.10/dist-packages/_pytest/stepwise.py:52: PytestCacheWarning: cache could not write path /home/vagrant/qa/tests/integration/.pytest_cache/v/cache/stepwise
    session.config.cache.set(STEPWISE_CACHE_DIR, [])

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===== 10 passed, 1 skipped, 88 deselected, 2 warnings in 212.99s (0:03:32) =====

Tier 1

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 99 items / 98 deselected / 1 selected

test_only_logs_after.py .                                                [100%]

================ 1 passed, 98 deselected, 2 warnings in 36.28s =================

nico-stefani · 2023-02-09T20:15:10Z

NLB Results

Tier 0

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 99 items / 88 deselected / 11 selected

test_basic.py .                                                          [  9%]
test_discard_regex.py .                                                  [ 18%]
test_only_logs_after.py ..                                               [ 36%]
test_path.py ...                                                         [ 63%]
test_regions.py ...                                                      [ 90%]
test_remove_from_bucket.py s                                             [100%]

===== 10 passed, 1 skipped, 88 deselected, 2 warnings in 212.58s (0:03:32) =====

Tier 1

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 99 items / 98 deselected / 1 selected

test_only_logs_after.py .                                                [100%]

=============================== warnings summary ===============================
================ 1 passed, 98 deselected, 2 warnings in 33.54s =================

nico-stefani · 2023-02-09T20:22:37Z

KMS Results

Tier 0

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 99 items / 91 deselected / 8 selected

test_basic.py .                                                          [ 12%]
test_discard_regex.py .                                                  [ 25%]
test_only_logs_after.py ..                                               [ 50%]
test_path.py ...                                                         [ 87%]
test_remove_from_bucket.py .                                             [100%]

=============================== warnings summary ===============================
=========== 8 passed, 91 deselected, 2 warnings in 176.49s (0:02:56) ===========

Tier 1

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 99 items / 98 deselected / 1 selected

test_only_logs_after.py .                                                [100%]

=============================== warnings summary ===============================
================ 1 passed, 98 deselected, 2 warnings in 38.15s =================

nico-stefani · 2023-02-15T20:43:12Z

Macie Results

Tier 0

root@ubuntu-jammy:/home/vagrant/qa/tests/integration/test_aws# pytest -k macie --tier 0
================================================================================================ test session starts ================================================================================================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 24 items / 16 deselected / 8 selected

test_basic.py .                                                                                                                                                                                               [ 12%]
test_discard_regex.py .                                                                                                                                                                                       [ 25%]
test_only_logs_after.py ..                                                                                                                                                                                    [ 50%]
test_path.py ...                                                                                                                                                                                              [ 87%]
test_remove_from_bucket.py .                                                                                                                                                                                  [100%]

================================================================================================= warnings summary ==================================================================================================
../../../../../../usr/local/lib/python3.10/dist-packages/_pytest/cacheprovider.py:433
  /usr/local/lib/python3.10/dist-packages/_pytest/cacheprovider.py:433: PytestCacheWarning: cache could not write path /home/vagrant/qa/tests/integration/.pytest_cache/v/cache/nodeids
    config.cache.set("cache/nodeids", sorted(self.cached_nodeids))

../../../../../../usr/local/lib/python3.10/dist-packages/_pytest/stepwise.py:52
  /usr/local/lib/python3.10/dist-packages/_pytest/stepwise.py:52: PytestCacheWarning: cache could not write path /home/vagrant/qa/tests/integration/.pytest_cache/v/cache/stepwise
    session.config.cache.set(STEPWISE_CACHE_DIR, [])

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================================= 8 passed, 16 deselected, 2 warnings in 164.31s (0:02:44) ==============================================================================

Tier 1

================================================================================================ test session starts ================================================================================================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 126 items / 125 deselected / 1 selected

test_only_logs_after.py .                                                                                                                                                                                     [100%]

================================================================================================= warnings summary ==================================================================================================
../../../../../../usr/local/lib/python3.10/dist-packages/_pytest/cacheprovider.py:433
  /usr/local/lib/python3.10/dist-packages/_pytest/cacheprovider.py:433: PytestCacheWarning: cache could not write path /home/vagrant/qa/tests/integration/.pytest_cache/v/cache/nodeids
    config.cache.set("cache/nodeids", sorted(self.cached_nodeids))

../../../../../../usr/local/lib/python3.10/dist-packages/_pytest/stepwise.py:52
  /usr/local/lib/python3.10/dist-packages/_pytest/stepwise.py:52: PytestCacheWarning: cache could not write path /home/vagrant/qa/tests/integration/.pytest_cache/v/cache/stepwise
    session.config.cache.set(STEPWISE_CACHE_DIR, [])

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================================== 1 passed, 125 deselected, 2 warnings in 36.87s ===================================================================================

nico-stefani · 2023-02-16T20:17:40Z

Trusted Advisor Results

Tier 0

root@ubuntu-jammy:/home/vagrant/qa/tests/integration/test_aws# pytest -k trusted --tier 0
================================================================================================ test session starts ================================================================================================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 24 items / 16 deselected / 8 selected

test_basic.py .                                                                                                                                                                                               [ 12%]
test_discard_regex.py .                                                                                                                                                                                       [ 25%]
test_only_logs_after.py ..                                                                                                                                                                                    [ 50%]
test_path.py ...                                                                                                                                                                                              [ 87%]
test_remove_from_bucket.py .                                                                                                                                                                                  [100%]

================================================================================================= warnings summary ==================================================================================================
../../../../../../usr/local/lib/python3.10/dist-packages/_pytest/cacheprovider.py:433
  /usr/local/lib/python3.10/dist-packages/_pytest/cacheprovider.py:433: PytestCacheWarning: cache could not write path /home/vagrant/qa/tests/integration/.pytest_cache/v/cache/nodeids
    config.cache.set("cache/nodeids", sorted(self.cached_nodeids))

../../../../../../usr/local/lib/python3.10/dist-packages/_pytest/stepwise.py:52
  /usr/local/lib/python3.10/dist-packages/_pytest/stepwise.py:52: PytestCacheWarning: cache could not write path /home/vagrant/qa/tests/integration/.pytest_cache/v/cache/stepwise
    session.config.cache.set(STEPWISE_CACHE_DIR, [])

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================================= 8 passed, 16 deselected, 2 warnings in 171.01s (0:02:51) ==============================================================================

Tier 1

root@ubuntu-jammy:/home/vagrant/qa/tests/integration/test_aws# pytest --tier 1 -k trusted
================================================================================================ test session starts ================================================================================================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 126 items / 125 deselected / 1 selected

test_only_logs_after.py .                                                                                                                                                                                     [100%]

================================================================================================= warnings summary ==================================================================================================
../../../../../../usr/local/lib/python3.10/dist-packages/_pytest/cacheprovider.py:433
  /usr/local/lib/python3.10/dist-packages/_pytest/cacheprovider.py:433: PytestCacheWarning: cache could not write path /home/vagrant/qa/tests/integration/.pytest_cache/v/cache/nodeids
    config.cache.set("cache/nodeids", sorted(self.cached_nodeids))

../../../../../../usr/local/lib/python3.10/dist-packages/_pytest/stepwise.py:52
  /usr/local/lib/python3.10/dist-packages/_pytest/stepwise.py:52: PytestCacheWarning: cache could not write path /home/vagrant/qa/tests/integration/.pytest_cache/v/cache/stepwise
    session.config.cache.set(STEPWISE_CACHE_DIR, [])

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================================== 1 passed, 125 deselected, 2 warnings in 34.30s ===================================================================================

nico-stefani · 2023-02-16T20:19:06Z

Guard Duty Results

Tier 0

root@ubuntu-jammy:/home/vagrant/qa/tests/integration/test_aws# pytest -k 'guardduty' --tier 0
============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 33 items / 17 deselected / 16 selected

test_basic.py ..                                                         [ 12%]
test_discard_regex.py ..                                                 [ 25%]
test_only_logs_after.py ....                                             [ 50%]
test_path.py ......                                                      [ 87%]
test_remove_from_bucket.py ..                                            [100%]

=============================== warnings summary ===============================
../../../../../../usr/local/lib/python3.10/dist-packages/_pytest/cacheprovider.py:433
  /usr/local/lib/python3.10/dist-packages/_pytest/cacheprovider.py:433: PytestCacheWarning: cache could not write path /home/vagrant/qa/tests/integration/.pytest_cache/v/cache/nodeids
    config.cache.set("cache/nodeids", sorted(self.cached_nodeids))

../../../../../../usr/local/lib/python3.10/dist-packages/_pytest/stepwise.py:52
  /usr/local/lib/python3.10/dist-packages/_pytest/stepwise.py:52: PytestCacheWarning: cache could not write path /home/vagrant/qa/tests/integration/.pytest_cache/v/cache/stepwise
    session.config.cache.set(STEPWISE_CACHE_DIR, [])

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========== 16 passed, 17 deselected, 2 warnings in 333.61s (0:05:33) ===========

Tier 1

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 33 items / 31 deselected / 2 selected

test_only_logs_after.py FF                                               [100%]

=================================== FAILURES ===================================
________ test_multiple_calls[guardduty_only_logs_after_multiple_calls] _________
------------------------------ Captured log call -------------------------------
DEBUG    wazuh_testing:cli_utils.py:23 Calling AWS module with: '[PosixPath('/var/ossec/wodles/aws/aws-s3'), '--bucket', 'wazuh-guardduty-integration-tests', '--type', 'guardduty', '--regions', 'us-east-1', '--aws_profile', 'qa', '--debug', '2']'
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Debug mode on - Level: 2
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Table does not exist; create
DEBUG    wazuh_testing:cli_utils.py:52 The functionality to process GuardDuty logs stored in S3 via Kinesis was deprecated in 4.5. Consider configuring GuardDuty to store its findings directly in an S3 bucket instead. Check https://documentation.wazuh.com/current/amazon/services/supported-services/guardduty.html for more information. 
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Marker: 2023/02/02
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ DB Maintenance
ERROR    wazuh_testing:cli_utils.py:62 Some logs may were processed
ERROR    wazuh_testing:cli_utils.py:63 Results found: 0
ERROR    wazuh_testing:cli_utils.py:64 Results expected: 1
_____ test_multiple_calls[native_guardduty_only_logs_after_multiple_calls] _____
------------------------------ Captured log call -------------------------------
DEBUG    wazuh_testing:cli_utils.py:23 Calling AWS module with: '[PosixPath('/var/ossec/wodles/aws/aws-s3'), '--bucket', 'wazuh-native-guardduty-integration-tests', '--type', 'guardduty', '--regions', 'us-east-1', '--aws_profile', 'qa', '--debug', '2']'
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Debug mode on - Level: 2
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Table does not exist; create
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Working on 819751203818 - us-east-1
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Marker: AWSLogs/819751203818/GuardDuty/us-east-1/2023/02/02
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ No logs to process in bucket: 819751203818/us-east-1
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ DB Maintenance
DEBUG    wazuh_testing:cli_utils.py:23 Calling AWS module with: '[PosixPath('/var/ossec/wodles/aws/aws-s3'), '--bucket', 'wazuh-native-guardduty-integration-tests', '--type', 'guardduty', '--regions', 'us-east-1', '--aws_profile', 'qa', '--debug', '2', '--only_logs_after', '2022-NOV-20']'
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Debug mode on - Level: 2
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Working on 819751203818 - us-east-1
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Marker: AWSLogs/819751203818/GuardDuty/us-east-1/2022/11/20
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: ++ Found new log: AWSLogs/819751203818/GuardDuty/us-east-1/2022/11/21/ukkce94f-eb7e-4f45-b324-834a0d0cas43.jsonl.gz
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: ++ Found new log: AWSLogs/819751203818/GuardDuty/us-east-1/2022/11/23/465ce94f-eb7e-4f45-b324-834a0d0cas43.jsonl.gz
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: ++ Found new log: AWSLogs/819751203818/GuardDuty/us-east-1/2022/11/26/987ce94f-eb7e-4f45-b324-834a0d0ccd08.jsonl.gz
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ DB Maintenance
DEBUG    wazuh_testing:cli_utils.py:23 Calling AWS module with: '[PosixPath('/var/ossec/wodles/aws/aws-s3'), '--bucket', 'wazuh-native-guardduty-integration-tests', '--type', 'guardduty', '--regions', 'us-east-1', '--aws_profile', 'qa', '--debug', '2', '--only_logs_after', '2022-NOV-20']'
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Debug mode on - Level: 2
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Working on 819751203818 - us-east-1
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Marker: AWSLogs/819751203818/GuardDuty/us-east-1/2022/11/20
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: ++ Skipping previously processed file: AWSLogs/819751203818/GuardDuty/us-east-1/2022/11/21/ukkce94f-eb7e-4f45-b324-834a0d0cas43.jsonl.gz
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: ++ Skipping previously processed file: AWSLogs/819751203818/GuardDuty/us-east-1/2022/11/23/465ce94f-eb7e-4f45-b324-834a0d0cas43.jsonl.gz
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: ++ Skipping previously processed file: AWSLogs/819751203818/GuardDuty/us-east-1/2022/11/26/987ce94f-eb7e-4f45-b324-834a0d0ccd08.jsonl.gz
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ DB Maintenance
DEBUG    wazuh_testing:cli_utils.py:23 Calling AWS module with: '[PosixPath('/var/ossec/wodles/aws/aws-s3'), '--bucket', 'wazuh-native-guardduty-integration-tests', '--type', 'guardduty', '--regions', 'us-east-1', '--aws_profile', 'qa', '--debug', '2', '--only_logs_after', '2022-NOV-22']'
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Debug mode on - Level: 2
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Working on 819751203818 - us-east-1
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Marker: AWSLogs/819751203818/GuardDuty/us-east-1/2022/11/22
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: ++ Skipping previously processed file: AWSLogs/819751203818/GuardDuty/us-east-1/2022/11/23/465ce94f-eb7e-4f45-b324-834a0d0cas43.jsonl.gz
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: ++ Skipping previously processed file: AWSLogs/819751203818/GuardDuty/us-east-1/2022/11/26/987ce94f-eb7e-4f45-b324-834a0d0ccd08.jsonl.gz
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ DB Maintenance
DEBUG    wazuh_testing:cli_utils.py:23 Calling AWS module with: '[PosixPath('/var/ossec/wodles/aws/aws-s3'), '--bucket', 'wazuh-native-guardduty-integration-tests', '--type', 'guardduty', '--regions', 'us-east-1', '--aws_profile', 'qa', '--debug', '2']'
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Debug mode on - Level: 2
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Working on 819751203818 - us-east-1
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Marker: AWSLogs/819751203818/GuardDuty/us-east-1/2023/02/02
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: ++ Found new log: AWSLogs/819751203818/GuardDuty/us-east-1/2023/02/02/dcf07e79-60ab-4e93-a3ca-e4a5a9eda2e9.jsonl.gz
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ DB Maintenance
ERROR    wazuh_testing:cli_utils.py:62 The AWS module did not use the correct marker
ERROR    wazuh_testing:cli_utils.py:63 Results found: 0
ERROR    wazuh_testing:cli_utils.py:64 Results expected: 1
=========================== short test summary info ============================
FAILED test_only_logs_after.py::test_multiple_calls[guardduty_only_logs_after_multiple_calls]
FAILED test_only_logs_after.py::test_multiple_calls[native_guardduty_only_logs_after_multiple_calls]
================ 2 failed, 31 deselected, 3 warnings in 55.90s =================

nico-stefani · 2023-02-24T20:32:50Z

WAF Results

Tier 0

root@ubuntu-jammy:/home/vagrant/qa/tests/integration/test_aws# pytest -k 'waf' --tier 0
================================================================================================ test session starts ================================================================================================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 24 items / 16 deselected / 8 selected

test_basic.py .                                                                                                                                                                                               [ 12%]
test_discard_regex.py .                                                                                                                                                                                       [ 25%]
test_only_logs_after.py ..                                                                                                                                                                                    [ 50%]
test_path.py ...                                                                                                                                                                                              [ 87%]
test_remove_from_bucket.py .                                                                                                                                                                                  [100%]

================================================================================================= warnings summary ==================================================================================================
../../../../../../usr/local/lib/python3.10/dist-packages/_pytest/cacheprovider.py:433
  /usr/local/lib/python3.10/dist-packages/_pytest/cacheprovider.py:433: PytestCacheWarning: cache could not write path /home/vagrant/qa/tests/integration/.pytest_cache/v/cache/nodeids
    config.cache.set("cache/nodeids", sorted(self.cached_nodeids))

../../../../../../usr/local/lib/python3.10/dist-packages/_pytest/stepwise.py:52
  /usr/local/lib/python3.10/dist-packages/_pytest/stepwise.py:52: PytestCacheWarning: cache could not write path /home/vagrant/qa/tests/integration/.pytest_cache/v/cache/stepwise
    session.config.cache.set(STEPWISE_CACHE_DIR, [])

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================================= 8 passed, 16 deselected, 2 warnings in 172.80s (0:02:52) ==============================================================================

Tier 1

root@ubuntu-jammy:/home/vagrant/qa/tests/integration/test_aws# pytest -k waf --disable-warnings --tier 1 --tb=no
================================================================================================ test session starts ================================================================================================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 195 items / 194 deselected / 1 selected

test_only_logs_after.py F                                                                                                                                                                                     [100%]

============================================================================================== short test summary info ==============================================================================================
FAILED test_only_logs_after.py::test_bucket_multiple_calls[waf_only_logs_after_multiple_calls] - wazuh_testing.modules.aws.cli_utils.OutputAnalysisError: The AWS module did not use the correct marker
================================================================================== 1 failed, 194 deselected, 3 warnings in 35.06s ===================================================================================

nico-stefani · 2023-02-24T20:34:52Z

Server Access Results

Tier 0

root@ubuntu-jammy:/home/vagrant/qa/tests/integration/test_aws# pytest -k server_access --disable-warnings --tier 0 --tb=no
================================================================================================ test session starts ================================================================================================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 195 items / 187 deselected / 8 selected

test_basic.py .                                                                                                                                                                                               [ 12%]
test_discard_regex.py .                                                                                                                                                                                       [ 25%]
test_only_logs_after.py ..                                                                                                                                                                                    [ 50%]
test_path.py ...                                                                                                                                                                                              [ 87%]
test_remove_from_bucket.py .                                                                                                                                                                                  [100%]

============================================================================= 8 passed, 187 deselected, 2 warnings in 173.15s (0:02:53) =============================================================================

Tier 1

root@ubuntu-jammy:/home/vagrant/qa/tests/integration/test_aws# pytest -k server_access --disable-warnings --tier 1 --tb=no
================================================================================================ test session starts ================================================================================================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 195 items / 194 deselected / 1 selected

test_only_logs_after.py F                                                                                                                                                                                     [100%]

============================================================================================== short test summary info ==============================================================================================
FAILED test_only_logs_after.py::test_bucket_multiple_calls[server_access_only_logs_after_multiple_calls] - wazuh_testing.modules.aws.cli_utils.OutputAnalysisError: The AWS module did not use the correct marker
================================================================================== 1 failed, 194 deselected, 3 warnings in 34.67s ===================================================================================

nico-stefani · 2023-02-28T14:26:38Z

Inspector Results

Tier 0

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 21 items / 16 deselected / 5 selected

test_basic.py .                                                          [ 20%]
test_only_logs_after.py .                                                [ 40%]
test_regions.py ..F                                                      [100%]

=================================== FAILURES ===================================
______________ test_service_regions[inspector_inexistent_region] _______________

------------------------------ Captured log call -------------------------------
ERROR    wazuh_testing:monitoring.py:465 The AWS module did not show correct message non-existent region
ERROR    wazuh_testing:monitoring.py:466 Results accumulated: 0
ERROR    wazuh_testing:monitoring.py:468 Results expected: 1
=========================== short test summary info ============================
FAILED test_regions.py::test_service_regions[inspector_inexistent_region] - T...
====== 1 failed, 4 passed, 16 deselected, 3 warnings in 125.92s (0:02:05) ======

Tier 1

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 21 items / 20 deselected / 1 selected

test_only_logs_after.py F                                                [100%]

=================================== FAILURES ===================================
____ test_service_multiple_calls[inspector_only_logs_after_multiple_calls] _____
------------------------------ Captured log call -------------------------------
DEBUG    wazuh_testing:cli_utils.py:23 Calling AWS module with: '[PosixPath('/var/ossec/wodles/aws/aws-s3'), '--service', 'inspector', '--regions', 'us-east-1', '--aws_profile', 'qa', '--debug', '2']'
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Debug mode on - Level: 2
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Getting alerts from "us-east-1" region.
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Table does not exist; create
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Listing findings starting from 2023-01-31 00:00:00
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ There are no new events in the "us-east-1" region
DEBUG    wazuh_testing:cli_utils.py:23 Calling AWS module with: '[PosixPath('/var/ossec/wodles/aws/aws-s3'), '--service', 'inspector', '--regions', 'us-east-1', '--aws_profile', 'qa', '--debug', '2', '--only_logs_after', '2023-JAN-30']'
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Debug mode on - Level: 2
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Getting alerts from "us-east-1" region.
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Listing findings starting from 2023-01-31 12:29:52.336149
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ There are no new events in the "us-east-1" region
ERROR    wazuh_testing:cli_utils.py:62 The AWS module did not process the expected number of events
ERROR    wazuh_testing:cli_utils.py:63 Results found: 0
ERROR    wazuh_testing:cli_utils.py:64 Results expected: 1
=========================== short test summary info ============================
FAILED test_only_logs_after.py::test_service_multiple_calls[inspector_only_logs_after_multiple_calls]
================ 1 failed, 20 deselected, 3 warnings in 28.31s =================

nico-stefani · 2023-03-06T19:48:56Z

Cloudwatch Results

Tier 0 results

================================================================================================ test session starts ================================================================================================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 195 items / 186 deselected / 9 selected

test_basic.py .                                                                                                                                                                                               [ 11%]
test_log_groups.py ..                                                                                                                                                                                         [ 33%]
test_only_logs_after.py ..                                                                                                                                                                                    [ 55%]
test_regions.py ..F                                                                                                                                                                                           [ 88%]
test_remove_from_bucket.py .                                                                                                                                                                                  [100%]

===================================================================================================== FAILURES ======================================================================================================
/home/vagrant/qa/deps/wazuh_testing/wazuh_testing/tools/monitoring.py:469: TimeoutError: The AWS module did not show correct message non-existent region
============================================================================================== short test summary info ==============================================================================================
FAILED test_regions.py::test_service_regions[cloudwatchlogs_inexistent_region] - TimeoutError: The AWS module did not show correct message non-existent region
======================================================================== 1 failed, 8 passed, 186 deselected, 3 warnings in 198.25s (0:03:18) ========================================================================

Tier 1

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 25 items / 24 deselected / 1 selected

test_only_logs_after.py .                                                [100%]
================ 1 passed, 24 deselected, 2 warnings in 41.45s =================

nico-stefani · 2023-03-06T19:51:15Z

Umbrella Results

Tier 0

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 24 items / 16 deselected / 8 selected

test_basic.py .                                                          [ 12%]
test_discard_regex.py .                                                  [ 25%]
test_only_logs_after.py ..                                               [ 50%]
test_path.py ...                                                         [ 87%]
test_remove_from_bucket.py .                                             [100%]
=========== 8 passed, 16 deselected, 2 warnings in 174.76s (0:02:54) ===========

Tier 1 results

root@ubuntu-jammy:/home/vagrant/qa/tests/integration/test_aws# pytest -k umbrella --disable-warnings --tier 1 --tb=line
================================================================================================ test session starts ================================================================================================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 195 items / 194 deselected / 1 selected

test_only_logs_after.py F                                                                                                                                                                                     [100%]

===================================================================================================== FAILURES ======================================================================================================
/home/vagrant/qa/deps/wazuh_testing/wazuh_testing/modules/aws/cli_utils.py:65: wazuh_testing.modules.aws.cli_utils.OutputAnalysisError: The AWS module did not use the correct marker
============================================================================================== short test summary info ==============================================================================================
FAILED test_only_logs_after.py::test_bucket_multiple_calls[cisco_umbrella_only_logs_after_multiple_calls] - wazuh_testing.modules.aws.cli_utils.OutputAnalysisError: The AWS module did not use the correct marker
================================================================================== 1 failed, 194 deselected, 3 warnings in 34.52s ===================================================================================

nico-stefani · 2023-03-14T16:10:52Z

Parser Results

Tier 0

root@ubuntu-jammy:/home/vagrant/qa/tests/integration/test_aws# pytest -k parser --disable-warnings --tier 0 --tb=line
================================================================================================ test session starts ================================================================================================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 195 items / 169 deselected / 26 selected

test_parser.py .....FFFF.FFF.F.FFF..FFF..                                                                                                                                                                     [100%]

===================================================================================================== FAILURES ======================================================================================================
============================================================================================== short test summary info ==============================================================================================
FAILED test_parser.py::test_empty_values_in_bucket[parser_empty_only_logs_after_in_bucket] - TimeoutError: The AWS module did not show the expected message about empty value
FAILED test_parser.py::test_empty_values_in_bucket[parser_empty_regions_in_bucket] - TimeoutError: The AWS module did not show the expected message about empty value
FAILED test_parser.py::test_empty_values_in_bucket[parser_empty_path_in_bucket] - TimeoutError: The AWS module did not show the expected message about empty value
FAILED test_parser.py::test_empty_values_in_bucket[parser_empty_path_suffix_in_bucket] - TimeoutError: The AWS module did not show the expected message about empty value
FAILED test_parser.py::test_empty_values_in_service[parser_empty_log_groups_in_service] - TimeoutError: The AWS module did not show the expected message about empty value
FAILED test_parser.py::test_empty_values_in_service[parser_empty_only_logs_after_in_service] - TimeoutError: The AWS module did not show the expected message about empty value
FAILED test_parser.py::test_empty_values_in_service[parser_empty_regions_in_service] - TimeoutError: The AWS module did not show the expected message about empty value
FAILED test_parser.py::test_invalid_values_in_bucket[parser_invalid_name_in_bucket] - TimeoutError: The AWS module did not show the expected message about invalid value
FAILED test_parser.py::test_invalid_values_in_bucket[parser_invalid_regions_in_bucket] - TimeoutError: The AWS module did not show the expected message about invalid value
FAILED test_parser.py::test_invalid_values_in_bucket[parser_invalid_path_in_bucket] - TimeoutError: The AWS module did not show the expected message about invalid value
FAILED test_parser.py::test_invalid_values_in_bucket[parser_invalid_path_suffix_in_bucket] - TimeoutError: The AWS module did not show the expected message about invalid value
FAILED test_parser.py::test_invalid_values_in_service[parser_invalid_log_groups_in_service] - TimeoutError: The AWS module did not show the expected message about invalid value
FAILED test_parser.py::test_invalid_values_in_service[parser_invalid_only_logs_after_in_service] - TimeoutError: The AWS module did not show the expected message about invalid value
FAILED test_parser.py::test_invalid_values_in_service[parser_invalid_regions_in_service] - TimeoutError: The AWS module did not show the expected message about invalid value
======================================================================= 14 failed, 12 passed, 169 deselected, 3 warnings in 288.18s (0:04:48) =======================================================================

juliamagan

GJ!

There are some requested changes:

Exceptions should be in a separate file
Imports should go with the path from wazuh_testing
The type of the parameters and return should be only in the documentation, not in the function declaration.
Sort the imports and remove the unused ones
Quotation marks should follow the following format
Do not use f-string if there are no variables
Warnings are not allowed
The default scope of the fixures is function, so it is not necessary to indicate it
There are fixture created that are not used
Is it expected that the PATH will be empty in some configurations?
Try to use existing functions as much as possible
Some more specific comments have been added, but they apply to several cases although the comments have not been repeated.

CHANGELOG.md

deps/wazuh_testing/wazuh_testing/modules/aws/constants.py

deps/wazuh_testing/wazuh_testing/modules/aws/data_generator.py

tests/integration/test_aws/test_basic.py

tests/integration/test_aws/test_discard_regex.py

tests/integration/test_aws/test_only_logs_after.py

Co-authored-by: Julia Magán <80041853+juliamagan@users.noreply.github.com>

juliamagan

Just a few changes.

Remember to run the tests again once you have applied all the changes and update the results obtained.

tests/integration/test_aws/__init__.py

deps/wazuh_testing/wazuh_testing/modules/aws/s3_utils.py

Co-authored-by: Julia Magán <80041853+juliamagan@users.noreply.github.com>

nico-stefani · 2023-04-28T13:43:26Z

After doing the changes I ran the tests again and these are the results:

root@ip-172-31-82-0:/home/ubuntu/wazuh-qa/tests/integration/test_aws# pytest --disable-warnings --tb=line --tier 0 -x
================================================================================================ test session starts ================================================================================================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/ubuntu/wazuh-qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.4, html-3.1.1, testinfra-5.0.0
collected 195 items / 16 deselected / 179 selected

test_basic.py ................                                                                                                                                                                                [  8%]
test_discard_regex.py ..............                                                                                                                                                                          [ 16%]
test_log_groups.py ..                                                                                                                                                                                         [ 17%]
test_only_logs_after.py ...............................                                                                                                                                                       [ 35%]
test_parser.py ..........................                                                                                                                                                                     [ 49%]
test_path.py ..........................................                                                                                                                                                       [ 73%]
test_path_suffix.py .........                                                                                                                                                                                 [ 78%]
test_regions.py ........................                                                                                                                                                                      [ 91%]
test_remove_from_bucket.py ...sss.........                                                                                                                                                                    [100%]

============================================================================ 176 passed, 3 skipped, 16 deselected in 3607.33s (1:00:07) =============================================================================
root@ip-172-31-82-0:/home/ubuntu/wazuh-qa/tests/integration/test_aws# pytest --disable-warnings --tb=line --tier 1 -x
================================================================================================ test session starts ================================================================================================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/ubuntu/wazuh-qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.4, html-3.1.1, testinfra-5.0.0
collected 195 items / 179 deselected / 16 selected

test_only_logs_after.py ..............x.                                                                                                                                                                      [100%]

============================================================================= 15 passed, 179 deselected, 1 xfailed in 362.08s (0:06:02) =============================================================================

juliamagan

LGTM

nico-stefani self-assigned this Feb 9, 2023

nico-stefani linked an issue Feb 9, 2023 that may be closed by this pull request

Add AWS integration tests #3333

Closed

nico-stefani changed the base branch from master to 4.5 February 23, 2023 18:30

nico-stefani force-pushed the 3333-aws-integration-tests branch from 0ca8cda to 4560775 Compare February 23, 2023 18:32

nico-stefani force-pushed the 3333-aws-integration-tests branch 3 times, most recently from 37611a9 to 5cf113b Compare March 13, 2023 21:50

nico-stefani force-pushed the 3333-aws-integration-tests branch 2 times, most recently from 88650f6 to 7f3a3bc Compare March 17, 2023 11:57

nico-stefani marked this pull request as ready for review April 18, 2023 19:03

nico-stefani force-pushed the 3333-aws-integration-tests branch from be4f8f3 to ba14c6f Compare April 18, 2023 19:10

davidjiglesias requested a review from juliamagan April 24, 2023 08:48

juliamagan requested changes Apr 25, 2023

View reviewed changes

nico-stefani force-pushed the 3333-aws-integration-tests branch from ccb5a1c to d76f822 Compare April 25, 2023 20:05

nico-stefani and others added 17 commits April 26, 2023 10:43

fix(#3333): Marked inspector tier 1 test as xfail

2f03e8c

fix(#3333): Splitted inspector and cloudwatch logs

6c69bd8

fix(#3333): Timeouts adjustments

b07befa

fix(#3333): Fixed tests cases for cloudwatch tier 1 tests

de4c3f8

fix(#3333): Fixed linter issues

a874fde

docs(#3333): Added changelog

d946703

fix(#3333): Fixed order and format

812864e

fix(#3333): Moved constants to __init__.py

2bd6078

fix(#3333): Replaced wazuh_log_monitor with file_monitoring

e06dbf8

fix(#3333): Moved constants to wazuh_testing/__init__.py

4fe6b7f

fix(#3333): Moved constants to test_aws/__init__.py

052c306

fix(#3333): Improved docstring

e4986f7

fix(#3333): Moved exception to new file

e4ece17

fix(#3333): Improved fixtures and removed unused ones

2c77c6f

fix(#3333): Fixed quotation marks

c9cd989

fix(#3333): Removed type hints

bdf9568

fix(#3339): Apply suggestions from code review

71fd9fe

Co-authored-by: Julia Magán <80041853+juliamagan@users.noreply.github.com>

nico-stefani force-pushed the 3333-aws-integration-tests branch from 95fb495 to 71fd9fe Compare April 26, 2023 13:45

fix(#3333): Install boto3 only in linux

6c322fc

nico-stefani requested a review from juliamagan April 26, 2023 15:07

juliamagan requested changes Apr 28, 2023

View reviewed changes

tests/integration/test_aws/__init__.py Outdated Show resolved Hide resolved

deps/wazuh_testing/wazuh_testing/modules/aws/s3_utils.py Outdated Show resolved Hide resolved

nico-stefani and others added 2 commits April 28, 2023 09:11

fix(#3339): Apply suggestions from code review

e7e6473

Co-authored-by: Julia Magán <80041853+juliamagan@users.noreply.github.com>

fix(#3333): Moved constants

5595884

nico-stefani force-pushed the 3333-aws-integration-tests branch from 096a144 to 5595884 Compare April 28, 2023 12:13

nico-stefani requested a review from juliamagan April 28, 2023 13:47

juliamagan approved these changes Apr 28, 2023

View reviewed changes

juliamagan merged commit 87a6bde into 4.5 Apr 28, 2023

juliamagan deleted the 3333-aws-integration-tests branch April 28, 2023 14:46

Rebits mentioned this pull request Jul 20, 2023

Move 4.5.0 CHANGELOG.md changes to 4.6.0 #4331

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AWS integration tests #3911

AWS integration tests #3911

nico-stefani commented Feb 8, 2023 •

edited

Loading

nico-stefani commented Feb 8, 2023 •

edited

Loading

nico-stefani commented Feb 9, 2023 •

edited

Loading

nico-stefani commented Feb 9, 2023 •

edited

Loading

nico-stefani commented Feb 9, 2023 •

edited

Loading

nico-stefani commented Feb 9, 2023 •

edited

Loading

nico-stefani commented Feb 9, 2023 •

edited

Loading

nico-stefani commented Feb 9, 2023

nico-stefani commented Feb 15, 2023 •

edited

Loading

nico-stefani commented Feb 16, 2023 •

edited

Loading

nico-stefani commented Feb 16, 2023 •

edited

Loading

nico-stefani commented Feb 24, 2023 •

edited

Loading

nico-stefani commented Feb 24, 2023 •

edited

Loading

nico-stefani commented Feb 28, 2023

nico-stefani commented Mar 6, 2023 •

edited

Loading

nico-stefani commented Mar 6, 2023 •

edited

Loading

nico-stefani commented Mar 14, 2023

juliamagan left a comment

juliamagan left a comment

nico-stefani commented Apr 28, 2023

juliamagan left a comment

AWS integration tests #3911

AWS integration tests #3911

Conversation

nico-stefani commented Feb 8, 2023 • edited Loading

Description

Added

Testing performed

Tier 0

Tier 1

nico-stefani commented Feb 8, 2023 • edited Loading

CloudTrail Results

Tier 0

Tier 1

nico-stefani commented Feb 9, 2023 • edited Loading

VPC Results

Tier 0

Tier 1

nico-stefani commented Feb 9, 2023 • edited Loading

Config Results

Tier 0

Tier 1

nico-stefani commented Feb 9, 2023 • edited Loading

ALB Results

Tier 0

Tier 1

nico-stefani commented Feb 9, 2023 • edited Loading

CLB Results

Tier 0

Tier 1

nico-stefani commented Feb 9, 2023 • edited Loading

NLB Results

Tier 0

Tier 1

nico-stefani commented Feb 9, 2023

KMS Results

Tier 0

Tier 1

nico-stefani commented Feb 15, 2023 • edited Loading

Macie Results

Tier 0

Tier 1

nico-stefani commented Feb 16, 2023 • edited Loading

Trusted Advisor Results

Tier 0

Tier 1

nico-stefani commented Feb 16, 2023 • edited Loading

Guard Duty Results

Tier 0

Tier 1

nico-stefani commented Feb 24, 2023 • edited Loading

WAF Results

Tier 0

Tier 1

nico-stefani commented Feb 24, 2023 • edited Loading

Server Access Results

Tier 0

Tier 1

nico-stefani commented Feb 28, 2023

Inspector Results

Tier 0

Tier 1

nico-stefani commented Mar 6, 2023 • edited Loading

Cloudwatch Results

Tier 0 results

Tier 1

nico-stefani commented Mar 6, 2023 • edited Loading

Umbrella Results

Tier 0

Tier 1 results

nico-stefani commented Mar 14, 2023

Parser Results

Tier 0

juliamagan left a comment

Choose a reason for hiding this comment

juliamagan left a comment

Choose a reason for hiding this comment

nico-stefani commented Apr 28, 2023

juliamagan left a comment

Choose a reason for hiding this comment

nico-stefani commented Feb 8, 2023 •

edited

Loading

nico-stefani commented Feb 8, 2023 •

edited

Loading

nico-stefani commented Feb 9, 2023 •

edited

Loading

nico-stefani commented Feb 9, 2023 •

edited

Loading

nico-stefani commented Feb 9, 2023 •

edited

Loading

nico-stefani commented Feb 9, 2023 •

edited

Loading

nico-stefani commented Feb 9, 2023 •

edited

Loading

nico-stefani commented Feb 15, 2023 •

edited

Loading

nico-stefani commented Feb 16, 2023 •

edited

Loading

nico-stefani commented Feb 16, 2023 •

edited

Loading

nico-stefani commented Feb 24, 2023 •

edited

Loading

nico-stefani commented Feb 24, 2023 •

edited

Loading

nico-stefani commented Mar 6, 2023 •

edited

Loading

nico-stefani commented Mar 6, 2023 •

edited

Loading