Add Flatcar to end-to-end tests, install Pypy on test VMs, etc #2779

narrieta · 2023-03-07T17:34:04Z

This PR includes several changes related to adding Flatcar to the test automation:

Added Flatcar (x64 and ARM64) to the endorsed distros
Installed Pypy 3.7 to the test VMs and made it the default Python for the user executing the tests (now test code does not need to worry about the Python version on the test VM).
Added support to skip tests
Added support to copy tools used by the tests to the test VM
Fixed a couple of bugs in AgentTestSuiteCombinator

narrieta · 2023-03-07T18:15:30Z

tests_e2e/orchestrator/lib/agent_test_suite.py

@@ -44,10 +43,13 @@
 import makepkg
 from azurelinuxagent.common.version import AGENT_VERSION
 from tests_e2e.orchestrator.lib.agent_test_loader import TestSuiteInfo
+from tests_e2e.tests.lib.agent_test import TestSkipped


Tests can raise TestSkipped to skip a non-supported distro, etc

narrieta · 2023-03-07T18:18:04Z

tests_e2e/orchestrator/lib/agent_test_suite.py

-        self.__context.is_vhd = self._get_required_parameter(variables, "c_vhd") != ""
-        self.__context.image_name = f"{node.os.name}-vhd" if self.__context.is_vhd else self._get_required_parameter(variables, "c_name")
+        self.__context.is_vhd = self._get_optional_parameter(variables, "c_vhd") != ""
+        self.__context.image_name = f"{node.os.name}-vhd" if self.__context.is_vhd else self._get_required_parameter(variables, "c_env_name")


I added a "vm_name" parameter for existing VMs, so I renamed "c_name" to "c_env_name"

tests_e2e/orchestrator/lib/agent_test_suite.py

narrieta · 2023-03-07T18:20:35Z

tests_e2e/orchestrator/lib/agent_test_suite.py


            # Copy the tarball to the local logs directory
            remote_path = "/tmp/waagent-logs.tgz"
            local_path = self.context.log_path/'{0}.tgz'.format(self.context.image_name)
            self._log.info("Copying %s:%s to %s", self.context.node.name, remote_path, local_path)
-            self.context.node.shell.copy_back(remote_path, local_path)
+            self.context.ssh_client.copy(remote_path, local_path, remote_source=True)


i removed the references to LISA's SSH client and now I am using the Agent's

narrieta · 2023-03-07T18:21:56Z

tests_e2e/orchestrator/lib/agent_test_suite.py

        with _set_thread_name(self.context.image_name):  # The thread name is added to self._log
+            self._log.info(


I move this message here from the initialization function. We set the thread name for the log after the initialization, so it was missing from the log for this message

narrieta · 2023-03-07T18:25:44Z

tests_e2e/orchestrator/sample_runbooks/existing_vm.yml

@@ -31,6 +31,8 @@ variable:
  #
  # These variables identify the existing VM, and the user for SSH connections
  #
+  - name: cloud


forgot to add "cloud" in my previous PR

narrieta · 2023-03-07T18:27:29Z

tests_e2e/orchestrator/scripts/get-agent-pythonpath

@@ -0,0 +1,74 @@
+#!/usr/bin/env bash


this script can be used to set the PYTHONPATH for any Python (this PR uses it for Pypy) before invoking 'waagent' or importing any of its modules

narrieta · 2023-03-07T18:28:08Z

tests_e2e/orchestrator/scripts/get-waagent-path

@@ -0,0 +1,57 @@
+#!/usr/bin/env bash


in some distros (e.g. Flatcar) 'waagent' is not in PATH

narrieta · 2023-03-07T18:29:21Z

tests_e2e/orchestrator/scripts/unzip.py

@@ -0,0 +1,36 @@
+#!/usr/bin/env python3


we were using the "unzip" package to do this, but that package may not be available in all tested distros (e.g. Flatcar does not even have a package manager) so now we use this Python script to unzip.

narrieta · 2023-03-07T18:30:11Z

tests_e2e/test_suites/images.yml

@@ -57,18 +59,19 @@ image-sets:
 # '<Publisher>:<Offer>:<Sku>:<Version>'
 #
 images:
-#


removed this duplicated comment... these distros are not added to 'endorsed' yet, and there is already a TODO comment there

maddieford · 2023-03-08T23:18:46Z

tests_e2e/tests/bvts/vm_access.py

@@ -38,6 +38,9 @@

 class VmAccessBvt(AgentTest):
    def run(self):
+        if type(self._context.node.os).__name__ == 'CoreOs' and self._context.node.os.information.full_version.startswith('Flatcar'):


Should we keep a list of skipped tests by distro? Or add a TODO here so we know to remove this when VMAccess becomes supported on Flatcar.

good point, we already have 3 or so tests/test cases that we skip. i'll create work items for those

nagworld9

Now ssh_client is added to context, can we extend that to tests instead initializing on every test.

tests_e2e/orchestrator/lib/agent_test_suite.py

nagworld9 · 2023-03-08T23:29:40Z

tests_e2e/orchestrator/scripts/get-agent-pythonpath

+  exit 1
+fi
+
+$python -c 'import site; print(":".join(site.getsitepackages()))'


can you add example what is the path looks like? is this like /usr/bin/pyhton3.6 or something else?

Not sure a sample path helps. getsitepackages() returns the path for the system-wide python modules (as opposed to user-wide modules). The agent uses the system-wide path.

This is a sample. If you think it helps, I can add it the comment

$ python3 Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import site >>> site.getsitepackages() ['/usr/local/lib/python3.10/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.10/dist-packages'] >>>

The actual values can change a lot (try this on flatcar, for example)

is this agent module python path or new python version path that we installed?

that sample is for my dev machine

nagworld9 · 2023-03-08T23:37:16Z

tests_e2e/orchestrator/scripts/install-agent

+
+# Ensure that AutoUpdate is enabled. some distros, e.g. Flatcar, don't have a waagent.conf
+# but AutoUpdate defaults to True so there is no need to anything in that case.
+if [[ -e /etc/waagent.conf ]]; then


Don't we need to add in setup of agent for flatcar distro to have conf file? It may break for people who copy agent from source if some config should not be default value for this distro

The default values are hardcoded in the agent. Specific tests may need to create /etc/waagent.conf, but the overall infrastructure should not create it, since that is not the way actual VMs would look like.

BTW, currently VMAccess breaks on Flatcar because it has a hard dependency on /etc/waagent.conf; the extension needs to fix this dependency, since user-created VMs won't have it. If the AgentTestSuite created the conf file, we would be masking this bug.

But why flatcar vm don't have it? what is special about it? My suspicion is that our agent source setup itself don't have it. In general distro owners copy or look at our agent setup to bootstrap agent right.

That is the way they setup Flatcar and is perfectly valid. Flatcar is supposed to be immutable so I imagine they are not allowing for any configuration changes by default,

nagworld9 · 2023-03-08T23:53:34Z

tests_e2e/orchestrator/scripts/install-agent

 #
 echo "Verifying agent installation..."
+
+PYTHONPATH=$(get-agent-pythonpath)
+export PYTHONPATH


After this change, I don't have to look for agent python in agent update test . Which python should be agent python and also default python right?

If tests need to import agent modules, or invoke 'waagent', they need to do the same as this script

PYTHONPATH=$(get-agent-pythonpath) export PYTHONPATH

Let me think a little if we should also set PYTHONPATH in .bash_profile, and then tests would not need to worry about it. I can do that on a separate PR.

I think it should be fine to set PYTHONPATH in .bas_profile. I'll add that to my next PR

nagworld9 · 2023-03-08T23:54:02Z

tests_e2e/orchestrator/scripts/install-agent

 check-version() {
+  waagent=$(get-waagent-path)


This is also helps in agent update, thanks for the change

flatcar forced me to do it :)

tests_e2e/orchestrator/lib/agent_test_suite.py

nagworld9 · 2023-03-09T01:35:23Z

Now ssh_client is added to context, can we extend that to tests as well instead initializing on every test.

narrieta · 2023-03-09T14:58:06Z

@nagworld9

Now ssh_client is added to context, can we extend that to tests as well instead initializing on every test.

Not all tests need the ssh client, but I agree it may be useful. If you feel we need it, please go ahead and send a PR adding to AgentTest.

codecov · 2023-03-09T15:09:46Z

Codecov Report

Merging #2779 (7c17f5f) into develop (0903727) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop    #2779      +/-   ##
===========================================
- Coverage    71.99%   71.99%   -0.01%     
===========================================
  Files          104      104              
  Lines        15839    15839              
  Branches      2265     2265              
===========================================
- Hits         11404    11403       -1     
  Misses        3913     3913              
- Partials       522      523       +1

Impacted Files	Coverage Δ
azurelinuxagent/common/protocol/hostplugin.py	`87.50% <0.00%> (-0.27%)`	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

narrieta added 3 commits March 6, 2023 12:36

Add Flatcar, install Pypy, etc

ae479bd

pypy arm64

3a3ca50

add flatcar

2187d11

narrieta requested review from ZhidongPeng, nagworld9 and maddieford as code owners March 7, 2023 17:34

narrieta commented Mar 7, 2023

View reviewed changes

narrieta and others added 5 commits March 7, 2023 15:41

uncompress

1473ce6

Merge branch 'develop' into flatcar

19735a7

typos

c3ede70

py2

8a5eeea

py2

22a3a5e

maddieford previously approved these changes Mar 8, 2023

View reviewed changes

nagworld9 reviewed Mar 9, 2023

View reviewed changes

pylint

7c17f5f

narrieta dismissed maddieford’s stale review via 7c17f5f March 9, 2023 15:02

nagworld9 approved these changes Mar 9, 2023

View reviewed changes

narrieta merged commit 0583078 into Azure:develop Mar 9, 2023

narrieta deleted the flatcar branch March 9, 2023 19:45

bignay2000 mentioned this pull request Nov 15, 2023

[BUG] Flatcar Linux 3602.2.1 Boot Errors #2984

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Flatcar to end-to-end tests, install Pypy on test VMs, etc #2779

Add Flatcar to end-to-end tests, install Pypy on test VMs, etc #2779

narrieta commented Mar 7, 2023

narrieta Mar 7, 2023

narrieta Mar 7, 2023

narrieta Mar 7, 2023

narrieta Mar 7, 2023

narrieta Mar 7, 2023

narrieta Mar 7, 2023

narrieta Mar 7, 2023

narrieta Mar 7, 2023

narrieta Mar 7, 2023

maddieford Mar 8, 2023

narrieta Mar 8, 2023

nagworld9 left a comment

nagworld9 Mar 8, 2023

narrieta Mar 9, 2023

narrieta Mar 9, 2023

nagworld9 Mar 9, 2023

narrieta Mar 9, 2023

nagworld9 Mar 8, 2023

narrieta Mar 9, 2023 •

edited

Loading

nagworld9 Mar 9, 2023

narrieta Mar 9, 2023

nagworld9 Mar 8, 2023

narrieta Mar 9, 2023

narrieta Mar 9, 2023

nagworld9 Mar 8, 2023

narrieta Mar 9, 2023

nagworld9 commented Mar 9, 2023

narrieta commented Mar 9, 2023

codecov bot commented Mar 9, 2023 •

edited

Loading

		with _set_thread_name(self.context.image_name): # The thread name is added to self._log
		self._log.info(

Add Flatcar to end-to-end tests, install Pypy on test VMs, etc #2779

Add Flatcar to end-to-end tests, install Pypy on test VMs, etc #2779

Conversation

narrieta commented Mar 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nagworld9 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

narrieta Mar 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nagworld9 commented Mar 9, 2023

narrieta commented Mar 9, 2023

codecov bot commented Mar 9, 2023 • edited Loading

Codecov Report

narrieta Mar 9, 2023 •

edited

Loading

codecov bot commented Mar 9, 2023 •

edited

Loading