[VCDA-2588] Set up environment for RDE based tests (vmware#1070) (vmw…

…are#1155) * [VCDA-2588] Set up environment for RDE based tests (vmware#1070) * set up environment for RDE based CSE tests Signed-off-by: Aniruddha Shamasundar <aniruddha.9794@gmail.com> * Added server tests back Signed-off-by: Aniruddha Shamasundar <aniruddha.9794@gmail.com> * Clean up defined entity artifacts after the tests are complete Signed-off-by: Aniruddha Shamasundar <aniruddha.9794@gmail.com> * Server tests changes Signed-off-by: Aniruddha Shamasundar <aniruddha.9794@gmail.com> * Addressed review comments Signed-off-by: Aniruddha Shamasundar <aniruddha.9794@gmail.com> * Addressed review comments Signed-off-by: Aniruddha Shamasundar <aniruddha.9794@gmail.com> * Fix flake8 errors Signed-off-by: Aniruddha Shamasundar <aniruddha.9794@gmail.com> * Address review comments Signed-off-by: Aniruddha Shamasundar <aniruddha.9794@gmail.com> * Address review comment Signed-off-by: Aniruddha Shamasundar <aniruddha.9794@gmail.com> * Change default template Signed-off-by: Aniruddha Shamasundar <aniruddha.9794@gmail.com>
andrew-ni · Aug 23, 2021 · 87e65a7 · 87e65a7
1 parent a8a7c28
commit 87e65a7
Show file tree

Hide file tree

Showing 10 changed files with 1,327 additions and 40 deletions.
diff --git a/container_service_extension/mqi/mqtt_extension_manager.py b/container_service_extension/mqi/mqtt_extension_manager.py
@@ -322,6 +322,22 @@ def check_extension_exists(self, ext_urn_id):
             self._debug_logger.debug(last_response.text)
             return False
 
+    def is_extension_enabled(self, ext_urn_id):
+        """Check if the MQTT Extension is enabled.
+
+        :param str ext_urn_id: the extension urn id
+
+        :return: boolean indicating if extension is enabled
+        :rtype: bool
+        """
+        try:
+            mqtt_ext_obj = self.get_extension_response_body_by_urn(ext_urn_id)
+            return mqtt_ext_obj['enabled']
+        except requests.exceptions.HTTPError:
+            last_response = self._cloudapi_client.get_last_response()
+            self._debug_logger.debug(last_response.text)
+            return False
+
     def setup_extension_token(self, token_name, ext_name, ext_version,
                               ext_vendor, ext_urn_id):
         """Handle setting up a single extension token.

diff --git a/container_service_extension/system_test_framework/environment.py b/container_service_extension/system_test_framework/environment.py
diff --git a/system_tests_v2/README.md b/system_tests_v2/README.md
@@ -0,0 +1,141 @@
+# CSE Testing
+
+## Usage
+
+```bash
+$ pip install -r test-requirements.txt
+$ cd container-service-extension/system_tests
+
+# modify base_config.yaml (more info in next section)
+# set up vCD instance (org, ovdc, ovdc network)
+
+# Run all tests (either works)
+$ pytest
+$ python -m pytest
+
+# Run a test module
+$ pytest test_cse_server.py
+
+# Run a test function
+$ pytest test_cse_server.py::mytestfunction
+
+# Useful options
+$ pytest --exitfirst # -x stop after first failure
+$ pytest --maxfail=2 # stop after 2 failures
+$ pytest --verbose # -v increases testing verbosity
+$ pytest --capture=no # -s print all output during testing (tells pytest not to capture output)
+$ pytest --disable-warnings
+
+# Common use case (outputs to 'testlog')
+$ pytest --disable-warnings -x -v -s test_cse_server.py > testlog
+```
+
+## base_config.yaml
+
+Testers should fill out this config file with vCD instance details
+
+Options for **'test'** section:
+
+| key                  | description                                                                                                                                                                                                                                         | value type | possible values | default value |
+|-----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------|-----------------|---------------|
+| teardown_installation | - Affects **test_cse_server.py** <br> - If True, delete all installation entities (even on test failure) <br> - If False, do not delete installation entities (even on test failure) <br> - If omitted, defaults to True                            | bool       | True, False     | True          |
+| teardown_clusters     | - Affects **test_cse_client.py** <br> - If True, delete test cluster on test failure <br> - If False, do not delete test cluster on test failure <br> - If omitted, defaults to True <br> - Successful client tests will not leave clusters up <br> | bool       | True, False     | True          |
+| test_all_templates    | - Affects **test_cse_client.py** <br> - If True, tests cluster operations on all templates found <br> - If False, tests cluster operations only for 1st template found <br> - If omitted, defaults to False                                         | bool       | True, False     | False         |
+
+## Notes
+
+- Client tests (**test_cse_client.py**) require an cluster admin and cluster author user with the same username and password specified in the config file **vcd** section
+- Server tests (**test_cse_server.py**) require you to have a public/private SSH key (RSA)
+  - These keys should be at: `~/.ssh/id_rsa` and `~/.ssh/id_rsa.pub`
+  - Keys must not be password protected (to remove key password, use `ssh-keygen -p`)
+  - ssh-key help: <https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/>
+- More detailed information can be found in the module docstrings
+
+---
+
+## Writing Tests with Pytest and Click
+
+Before writing a test, first check **conftest.py** for any 'autouse=True' fixtures. Then check
+the module itself for any 'autouse=True' fixtures. When writing a test, any fixtures
+defined in **conftest.py** can be used. When creating new fixtures, place it in the module
+if its functionality is specific to that test module. If the functionality can be used
+across multiple test modules, then place it in **conftest.py**
+
+### Fixtures (setUp, tearDown)
+
+```python
+@pytest.fixture()
+def my_fixture():
+    print('my_fixture setup')
+    yield
+    print('my_fixture teardown')
+
+@pytest.fixture()
+def another_fixture():
+    important_variable = {'key': 'value'}
+    yield important_variable
+    print('another_fixture teardown')
+
+def my_test(my_fixture):
+    assert my_fixture is None
+
+def my_test_2(another_fixture):
+    assert isinstance(another_fixture, dict)
+    print(another_fixture['key'])
+```
+
+Common keyword arguments for @pytest.fixture()
+
+| keyword | description                                                | value type | possible values                          | default value |
+|---------|------------------------------------------------------------|------------|------------------------------------------|---------------|
+| scope   | defines when and how often the fixture should run          | str        | 'session', 'module', 'class', 'function' | 'function'    |
+| autouse | if True, fixture runs automatically with respect to @scope | bool       | True, False                              | False         |
+
+- Fixture teardown (after yield) executes even if test raises an exception (including AssertionError)
+
+- Shared fixtures should be defined in **conftest.py** according to pytest conventions. Fixtures defined here are autodiscovered by pytest and don't need to be imported.
+
+- Fixtures specific to a test module should be defined in the module directly.
+
+### Click's CLIRunner
+
+## Example
+
+```python
+import container_service_extension.system_test_framework.environment as env
+from container_service_extension.server_cli import cli
+from vcd_cli.vcd import vcd
+
+# test command: `cse sample --output myconfig.yaml`
+cmd = 'sample --output myconfig.yaml'
+result = env.CLI_RUNNER.invoke(cli, cmd.split(), catch_exceptions=False)
+# catch_exceptions=False tell Click not to swallow exceptions, so we can inspect if something went wrong
+assert result.exit_code == 0, f"Command [{cmd}] failed."
+
+# test command: `vcd cse template list`
+cmd = 'cse template list'
+result = env.CLI_RUNNER.invoke(vcd, cmd.split(), catch_exceptions=False)
+assert result.exit_code == 0, f"Command[{cmd}] failed."
+```
+
+These small tests using Click's `CLIRunner` and `invoke` function only validate command structure.
+These assert statements will pass because the commands themselves are valid, even if an error is thrown during the command execution.
+
+## Pytest logging mechanism
+
+pytest-logger is a plugin used to log during test execution. The plugin makes use of hooks to configure logs.
+The `pytest_logger.py` file contains the logger which can be imported in different test files.
+
+The hooks `pytest_logger_config(logger_config)` and `pytest_logger_logdirlink(config)` are configured in conftest.py to create
+a symlink `pytest_log` to log directory created by pytest. A different log file will be created for each test executed.
+
+To log information, please import and use the logger PYTEST_LOGGER defined in `pytest_logger.py` module.
+
+---
+
+### Helpful links
+
+- Usage and Invocations: <https://docs.pytest.org/en/latest/usage.html>
+- Fixtures: <https://docs.pytest.org/en/latest/fixture.html>
+- Click Testing: <http://click.palletsprojects.com/en/7.x/testing/>
+- Logging: <https://pypi.org/project/pytest-logger/>
diff --git a/system_tests_v2/__init__.py b/system_tests_v2/__init__.py
diff --git a/system_tests_v2/base_config.yaml b/system_tests_v2/base_config.yaml
@@ -0,0 +1,64 @@
+# Valid config file
+# Fill in fields marked with '???'
+
+test:
+  teardown_installation: true                              # Affects test_cse_server.py.
+                                                           #   if true, delete all installation entities (even on test failure).
+                                                           #   if false, do not delete installation entities (even on test success).
+                                                           #   if this key is omitted, defaults to true.
+  teardown_clusters: true                                  # Affects test_cse_client.py.
+                                                           #   if true, delete test cluster (env.TEST_CLUSTER_NAME) on test failure.
+                                                           #   if false, do not delete test cluster on test failure.
+                                                           #   if this key is omitted, defaults to true.
+                                                           # Successful client tests will not leave clusters up.
+  test_all_templates: true                                 # Affects test_cse_client.py.
+                                                           #   if true, tests cluster deployment on all templates found.
+                                                           #   if false, tests cluster deployment only for first template found.
+                                                           #   if this key is omitted, defaults to false.
+  test_templates: "???"                                    # Tests will only create these templates if test_all_templates is set to false
+                                                           #   format -> "template_1_name:template_1_revision,template_2_name:template_2_revision"
+  upgrade_template_repo_url: '???'                         #
+  network: '???'                                           # org network within @vdc that will be used during testing
+                                                           #   Should have outbound access to the public internet
+  org: '???'                                               # vCD org where all the test will be run
+  storage_profile: '???'                                   # name of the storage profile to use while creating clusters on this org vdc
+  vdc: '???'                                               # Org VDC powering the org
+
+mqtt:
+  verify_ssl: false
+
+vcd:
+  host: '???'
+  log: true
+  password: '???'
+  port: 443
+  username: '???'
+  verify: false
+
+vcs:
+- name: '???'
+  password: '???'
+  username: '???'
+  verify: false
+
+service:
+  enforce_authorization: false
+  processors: 5
+  log_wire: false
+  telemetry:
+    enable: false
+  legacy_mode: false
+
+broker:
+  catalog: cse                          # public shared catalog within org where the template will be published
+  default_template_name: ubuntu-16.04_k8-1.21_weave-2.8.1
+                                        # name of the default template to use if none is specified
+  default_template_revision: 1          # revision of the default template to use if none is specified
+  ip_allocation_mode: pool              # dhcp or pool
+  network: '???'                        # org network within @vdc that will be used during the install process to build the template
+                                        #   Should have outbound access to the public internet
+                                        #   CSE appliance doesn't need to be connected to this network
+  org: '???'                            # vCD org that contains the shared catalog where the master templates will be stored
+  remote_template_cookbook_url: http://raw.githubusercontent.com/vmware/container-service-extension-templates/dev/template_v2.yaml
+  storage_profile: '*'                  # name of the storage profile to use when creating the temporary vApp used to build the template
+  vdc: '???'                            # VDC within @org that will be used during the install process to build the template
diff --git a/system_tests_v2/conftest.py b/system_tests_v2/conftest.py
@@ -0,0 +1,124 @@
+# container-service-extension
+# Copyright (c) 2019 VMware, Inc. All Rights Reserved.
+# SPDX-License-Identifier: BSD-2-Clause
+
+"""
+conftest.py is used by pytest to automatically find shared fixtures.
+
+Fixtures defined here can be used without importing.
+"""
+import os
+
+import pytest
+import system_tests_v2.pytest_logger as pytest_logger
+
+import container_service_extension.system_test_framework.environment as env
+import container_service_extension.system_test_framework.utils as testutils
+
+
+def pytest_logger_config(logger_config):
+    # adds two loggers, which will:
+    # - log to pytest_logger at 'warn' level
+    logger_config.add_loggers([pytest_logger.PYTEST_LOGGER_NAME],
+                              stdout_level='warn')
+    # default --loggers option is set to log pytest_logger at WARN level
+    logger_config.set_log_option_default(pytest_logger.PYTEST_LOGGER_NAME)
+
+
+def pytest_logger_logdirlink(config):
+    return os.path.join(os.path.dirname(__file__), pytest_logger.PYTEST_LOG_FILE_NAME)  # noqa: E501
+
+
+@pytest.fixture(scope='session', autouse=True)
+def environment():
+    """Fixture to setup and teardown the session environment.
+
+    This fixture executes automatically for test session setup and teardown.
+    Does not have any side effects to vCD.
+
+    Setup tasks:
+    - initialize variables (org/vdc href, client, mqtt settings)
+    - create cluster admin and cluster author roles
+
+    Teardown tasks:
+    - logout client
+    """
+    env.init_rde_environment(logger=pytest_logger.PYTEST_LOGGER)
+    yield
+    env.cleanup_environment(logger=pytest_logger.PYTEST_LOGGER)
+
+
+@pytest.fixture(scope='session', autouse=True)
+def vcd_users():
+    """Fixture to setup required users if they do not exist already.
+
+    This fixture executes automatically for test session setup and teardown.
+    User credentials are in 'system_test_framework/environment.py'
+
+    Setup tasks:
+    - create Cluster admin user if it doesn't exist
+    - create Cluster author user if it doesn't exist
+    """
+    # org_admin -> cluster_admin
+    # k8_author -> cluster_author
+    env.create_user(env.CLUSTER_ADMIN_NAME,
+                    env.CLUSTER_ADMIN_PASSWORD,
+                    env.CLUSTER_ADMIN_ROLE_NAME,
+                    logger=pytest_logger.PYTEST_LOGGER)
+    env.create_user(env.CLUSTER_AUTHOR_NAME,
+                    env.CLUSTER_AUTHOR_PASSWORD,
+                    env.CLUSTER_AUTHOR_ROLE_NAME,
+                    logger=pytest_logger.PYTEST_LOGGER)
+
+
+@pytest.fixture
+def config():
+    """Fixture to setup and teardown an active config file.
+
+    Usage: add the parameter 'config' to the test function. This 'config'
+        parameter is the dict representation of the config file, and can be
+        used in the test function.
+
+    Tasks:
+    - create config dict from env.BASE_CONFIG_FILEPATH
+    - create active config file at env.ACTIVE_CONFIG_FILEPATH
+    - adjust active config file security
+
+    yields config dict
+    """
+    config = env.setup_active_config()
+    yield config
+    env.teardown_active_config()
+
+
+@pytest.fixture
+def test_config():
+    """Fixture to provide 'test' section of test config to individual tests."""
+    config = testutils.yaml_to_dict(env.BASE_CONFIG_FILEPATH)
+    return config['test']
+
+
+@pytest.fixture
+def publish_native_right_bundle():
+    """Publish CSE native right bundle to deployment org.
+
+    Usage: add parameter 'publish_native_right_bundle' to the test function.
+        This fixture will be executed after the test function completes
+
+    Tasks done:
+    - publish cse native right bundle to deployment org (org specified in
+        'test' section of base_config.yaml)
+    - assign appropriate rights to roles in test org
+    """
+    yield
+    env.publish_right_bundle_to_deployment_org(
+        logger=pytest_logger.PYTEST_LOGGER)
+    env.assign_native_rights(env.CLUSTER_ADMIN_ROLE_NAME,
+                             ["cse:nativeCluster: Full Access",
+                              "cse:nativeCluster: Modify",
+                              "cse:nativeCluster: View"],
+                             logger=pytest_logger.PYTEST_LOGGER)
+    env.assign_native_rights(env.CLUSTER_AUTHOR_ROLE_NAME,
+                             ["cse:nativeCluster: Modify",
+                              "cse:nativeCluster: View"],
+                             logger=pytest_logger.PYTEST_LOGGER)
diff --git a/system_tests_v2/pytest_logger.py b/system_tests_v2/pytest_logger.py
@@ -0,0 +1,10 @@
+# container-service-extension
+# Copyright (c) 2021 VMware, Inc. All Rights Reserved.
+# SPDX-License-Identifier: BSD-2-Clause
+
+import logging
+
+
+PYTEST_LOG_FILE_NAME = "pytest_log"
+PYTEST_LOGGER_NAME = "log"
+PYTEST_LOGGER: logging.Logger = logging.getLogger(PYTEST_LOGGER_NAME)