Add BVT for the agent #2719

narrieta · 2023-01-03T15:07:44Z

This PR includes a new agent-bvt test that replaces the current BVT in DCR. There is still a extension I need to add (LAD), but I will do that on a separate PR. The current BVT also includes other tests that are targeted towards provisioning. Those will be added as a different test suite at a later stage.

The PR also includes multiple improvements in logging, reporting test results, organization of the tests, etc. It also defines some of the patterns the tests will follow. I documented those in the test wiki and will also point them out as comments in this PR.

narrieta · 2023-01-03T15:11:59Z

.github/workflows/ci_pr.yml

@@ -56,7 +56,7 @@ jobs:
            PYLINTOPTS: "--rcfile=ci/3.6.pylintrc --ignore=tests_e2e"

          - python-version: 3.9
-            PYLINTOPTS: "--rcfile=ci/3.6.pylintrc --ignore=azure_models.py,BaseExtensionTestClass.py"


I removed those files, which were simple prototypes.

narrieta · 2023-01-03T15:12:44Z

makepkg.py

@@ -74,7 +74,8 @@ def run(agent_family, output_directory, log):
    os.makedirs(bin_path)
    log.info("Created {0} directory".format(target_path))

-    args = ["python3", "setup.py", "bdist_egg", "--dist-dir={0}".format(bin_path)]


This allows invoking makepkg without cd'ing to the WALinuxAgent directory

narrieta · 2023-01-03T15:15:53Z

tests_e2e/orchestrator/lib/agent_test_suite.py

+        return self._node
+
+
+class AgentTestSuite(TestSuite):


I derived AgentTestSuite fromLISA's TestSuite again, since it simplifies the calling pattern for tests and allows the use of some metadata. I moved all its data members into the AgentTestContext and replaced self._log with the global Logger.

narrieta · 2023-01-03T15:18:27Z

tests_e2e/orchestrator/lib/agent_test_suite.py

        """
+        self._set_context(node)


I improved the way we report test results. See the latest test run for an example.

narrieta · 2023-01-03T15:19:13Z

tests_e2e/orchestrator/scripts/run-scenarios

@@ -41,4 +41,4 @@ lisa \
  --log_path "$HOME/logs" \
  --working_path "$HOME/logs" \
  -v subscription_id:"$SUBSCRIPTION_ID" \
-  -v identity_file:"$HOME/.ssh/id_rsa.pub"


This should have been the private key, not the public key

narrieta · 2023-01-03T15:23:41Z

tests_e2e/scenarios/lib/logging_utils.py

@@ -1,33 +0,0 @@
-# Create a base class


replace by logging.py

narrieta · 2023-01-03T15:24:35Z

tests_e2e/scenarios/lib/models.py

@@ -1,135 +0,0 @@
-import os


No longer needed

narrieta · 2023-01-03T15:28:50Z

tests_e2e/scenarios/lib/retry.py

+        try:
+            return operation()
+        except Exception as e:
+            # TODO: Do we need to retry on msrestazure.azure_exceptions.CloudError?


Some of the logic in the prototype was retrying on CloudError and HttpResponseError.

The latter, though, is raised in perfectly valid scenarios (e.g. trying to retrieve the instance view of an extension that not installed, or passing an incorrect value to an extension parameter) and should not be retried (maybe some of its derived classes should be retried).

So far I have not seen any instances of CloudError, so I just added a TODO note here.

narrieta · 2023-01-03T15:31:56Z

tests_e2e/scenarios/lib/virtual_machine.py

+
+#
+# This module includes facilities to execute some operations on virtual machines and scale sets (list extensions, restart, etc).
+#


This module has not been tested and is not used currently. Similar functionality was implemented in the prototype's azure_models.py, and I decided to port it as well, since for sure some tests will need it.

narrieta · 2023-01-03T15:32:37Z

tests_e2e/scenarios/lib/vm_extension.py

+        ).wait(timeout=_TIMEOUT))
+
+
+class VmssExtension(_VmExtensionBaseClass):


This class has not been tested and is not used currently. Similar functionality was implemented in the prototype's azure_models.py, and I decided to port it as well, since for sure some tests will need it.

maddieford · 2023-01-03T20:35:07Z

tests_e2e/orchestrator/lib/agent_test_suite.py

        finally:
            self._clean_up()

+        # Fail the entire test suite if any test failed
+        assert_that(failed).described_as("One or more tests failed").is_length(0)


Do we do anything with the test names we're appending to the failed list? It looks like we're just checking the length of the list here, should we change failed to a counter?

The message in the assertion failure includes the list of list of tests and the count, so we'll have a summary of the tests that failed.

Also, we output them as part of 'results' in the summary (we append failed tests both to 'failed' and 'results')

2023-01-03T00:36:02Z.676 [INFO] **************************************** [Test Results] **************************************** 2023-01-03T00:36:02Z.676 [INFO] 2023-01-03T00:36:02Z.676 [INFO] [Passed] ExtensionOperationsBvt 2023-01-03T00:36:02Z.676 [INFO] [Failed] RunCommandBvt 2023-01-03T00:36:02Z.676 [INFO] [Passed] VmAccessBvt

maddieford · 2023-01-03T20:44:28Z

tests_e2e/orchestrator/lib/agent_test_suite.py

    def execute_script_on_node(self, script_path: Path, parameters: str = "", sudo: bool = False) -> int:
        """
        Executes the given script on the test node; if 'sudo' is True, the script is executed using the sudo command.
        """
        custom_script_builder = CustomScriptBuilder(script_path.parent, [script_path.name])
-        custom_script = self._node.tools[custom_script_builder]
+        custom_script = self.context.node.tools[custom_script_builder]

        if parameters == '':
            command_line = f"{script_path}"
        else:
            command_line = f"{script_path} {parameters}"


Do we need to check if parameters == ''? Can we get rid of the first case and just have an extra whitespace at the end of command_line?

Avoiding the extra space was intentional. A minor detail, but later on we surround the command line with [] and the extra space looks odd.

maddieford · 2023-01-03T20:53:26Z

tests_e2e/scenarios/lib/agent_test_context.py

+
+    class Paths:
+        # E1101: Instance of 'list' has no '_path' member (no-member)
+        DEFAULT_TEST_SOURCE_DIRECTORY = Path(tests_e2e.__path__._path[0])  # pylint: disable=E1101


# pylint: disable=E1101
Are these comments suppressing pylint warnings?

Yes, they are. I've been adding the warning as an extra comment one line above the problematic code. In this case the warning is "# E1101: Instance of 'list' has no '_path' member (no-member)"; somehow pylint seems to think path is a list (it is not)

maddieford · 2023-01-03T22:30:29Z

tests_e2e/scenarios/lib/logging.py

+    console_handler = logging.StreamHandler()
+    log.addHandler(console_handler)
+
+log .setLevel(logging.INFO)


extra space here? log .setLevel(logging.INFO)

Yes, will remove it, thanks

maddieford · 2023-01-03T22:31:15Z

tests_e2e/scenarios/lib/logging.py

+
+log: logging.Logger = logging.getLogger("lisa")
+
+if not log.hasHandlers():


What is hasHandlers() doing here?

Handlers do the actual log output. We attempt to check whether we are running under lisa by doing "logging.getLogger("lisa")".

If the "lisa" logger does not exist (i.e. the test is being run from the command-line, not from a LISA Test Suite), getLogger() will return a new Logger with no handlers. In this case, we throw away that Logger and create a new one named "waagent" and then add a handler to output to stdout.

maddieford · 2023-01-03T22:58:20Z

tests_e2e/scenarios/lib/shell.py

@@ -0,0 +1,53 @@
+#!/usr/bin/env python3


What will this module be used for/ what other utilities will we add to this?

It'll be used for shell-related functionality. At this point I do not anticipate other functionality that running a command.

maddieford · 2023-01-03T23:11:38Z

tests_e2e/scenarios/lib/vm_extension.py

+        Performs an enable operation on the extension.
+
+        NOTE: 'force_update' is not a parameter of the actual ARM API. It is provided for convenience: If set to True,
+              the 'force_update_tag' can be left unspecified and this method will generate a random tag.


What is the force_update_tag used for?

If the extension settings do not change, invoking the extension won't generate a new goal state (because the model has not changed). To work around that, CRP exposes the force_update_tag, which is an arbitrary string with no meaning other than providing a way to specify unique values on different calls to enable (our code uses GUIDs)

maddieford · 2023-01-03T23:17:01Z

tests_e2e/scenarios/tests/bvts/run_command.py

+# BVT for RunCommand.
+#
+# Note that there are two incarnations of RunCommand (which are actually two different extensions):
+# Microsoft.CPlat.Core.RunCommandHandlerLinux and Microsoft.OSTCExtensions.VMAccessForLinux. This


Should we change this
Microsoft.OSTCExtensions.VMAccessForLinux
to
Microsoft.OSTCExtensions.RunCommand

Yes, will update this, thanks

nagworld9 · 2023-01-05T00:55:23Z

tests_e2e/scenarios/lib/shell.py

+        self.stderr: str = stderr
+
+
+def run_command(command: Any, shell=False) -> str:


since it's retuning can we rename to run_command_output or something like that.

I think run_command is appropriate, the function is not different to other functions that return a value so adding "_output" is not needed.

nagworld9 · 2023-01-05T01:15:29Z

tests_e2e/orchestrator/lib/agent_test_suite.py

+                        result = f"[Failed] {test.__name__}"
+
+                        log.info("")
+                        log.exception("******************** %s\n", result)


can we log error msg too. I think test name is not enough for debugging.

log.exception logs the exception message and traceback

nagworld9 · 2023-01-05T01:16:49Z

tests_e2e/scenarios/lib/agent_test.py

+        except SystemExit:  # Bad arguments
+            pass
+        except:  # pylint: disable=bare-except
+            log.exception("Test failed")


can we add error msg too?

log.exception does that

codecov · 2023-01-05T15:02:52Z

Codecov Report

Merging #2719 (fe57b35) into develop (a3a41bd) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop    #2719   +/-   ##
========================================
  Coverage    71.96%   71.96%           
========================================
  Files          104      104           
  Lines        15765    15765           
  Branches      2244     2244           
========================================
  Hits         11345    11345           
  Misses        3909     3909           
  Partials       511      511

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

narrieta added 14 commits December 26, 2022 16:26

Add BVT for extension workflow

ffae470

logging

befcad3

update scenario

097bdf9

test context

c91be13

AgentTest

e05bef3

vmaccess bvt

8258a6a

arguments; protected settings

4d3cf88

fix username

7776ee6

RunCommand

24596f3

Test dependencies

ac119f9

ssh key mode

10a19bf

ssh key file

14b9ea0

key mode

7c9bb6c

log type

373c479

narrieta requested review from ZhidongPeng, nagworld9 and maddieford as code owners January 3, 2023 15:07

narrieta commented Jan 3, 2023

View reviewed changes

maddieford reviewed Jan 3, 2023

View reviewed changes

PR review

afe8ef6

maddieford previously approved these changes Jan 4, 2023

View reviewed changes

nagworld9 reviewed Jan 5, 2023

View reviewed changes

Unused import

fe57b35

narrieta dismissed maddieford’s stale review via fe57b35 January 5, 2023 14:57

maddieford approved these changes Jan 6, 2023

View reviewed changes

nagworld9 approved these changes Jan 6, 2023

View reviewed changes

narrieta merged commit 2bd03c9 into Azure:develop Jan 6, 2023

narrieta deleted the bvt branch January 6, 2023 17:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add BVT for the agent #2719

Add BVT for the agent #2719

narrieta commented Jan 3, 2023

narrieta Jan 3, 2023

narrieta Jan 3, 2023 •

edited

Loading

narrieta Jan 3, 2023 •

edited

Loading

narrieta Jan 3, 2023

narrieta Jan 3, 2023

narrieta Jan 3, 2023

narrieta Jan 3, 2023

narrieta Jan 3, 2023 •

edited

Loading

narrieta Jan 3, 2023

narrieta Jan 3, 2023

maddieford Jan 3, 2023

narrieta Jan 3, 2023

maddieford Jan 3, 2023

narrieta Jan 3, 2023

maddieford Jan 3, 2023

narrieta Jan 3, 2023

maddieford Jan 3, 2023

narrieta Jan 3, 2023

maddieford Jan 3, 2023

narrieta Jan 3, 2023 •

edited

Loading

maddieford Jan 3, 2023

narrieta Jan 3, 2023

maddieford Jan 3, 2023

narrieta Jan 3, 2023

maddieford Jan 3, 2023

narrieta Jan 3, 2023

nagworld9 Jan 5, 2023

narrieta Jan 5, 2023

nagworld9 Jan 5, 2023

narrieta Jan 5, 2023

nagworld9 Jan 5, 2023

narrieta Jan 5, 2023

codecov bot commented Jan 5, 2023

		).wait(timeout=_TIMEOUT))


		class VmssExtension(_VmExtensionBaseClass):


		log: logging.Logger = logging.getLogger("lisa")

		if not log.hasHandlers():

		self.stderr: str = stderr


		def run_command(command: Any, shell=False) -> str:

Add BVT for the agent #2719

Add BVT for the agent #2719

Conversation

narrieta commented Jan 3, 2023

Choose a reason for hiding this comment

narrieta Jan 3, 2023 • edited Loading

Choose a reason for hiding this comment

narrieta Jan 3, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

narrieta Jan 3, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

narrieta Jan 3, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jan 5, 2023

Codecov Report

narrieta Jan 3, 2023 •

edited

Loading

narrieta Jan 3, 2023 •

edited

Loading

narrieta Jan 3, 2023 •

edited

Loading

narrieta Jan 3, 2023 •

edited

Loading