Use common download logic for agent downloads #2682

narrieta · 2022-10-07T20:10:07Z

This PR completes the refactoring of the download logic to use a common implementation for all downloads.

The downloads on this PR are for the agent's package. I made one change of behavior: now we delete the ZIP file once we expand it (both for the agent and extensions). These ZIPs are not used after we expand them and we have received reports that the current cleanup logic does not always work. I did not remove the cleanup logic, since we still need to cleanup files downloaded by previous agents or the daemon.

Fixes #2489

narrieta · 2022-10-07T20:15:49Z

azurelinuxagent/common/protocol/goal_state.py

@@ -152,13 +152,6 @@ def _fetch_manifest(self, manifest_type, name, uris):
        except Exception as e:
            raise ProtocolError("Failed to retrieve {0} manifest. Error: {1}".format(manifest_type, ustr(e)))

-    def download_extension(self, uris, destination, on_downloaded=lambda: True):


Initially I added this method here for consistency with fetch_extension_manifest above, but it really doesn't quite fit here so I'm removing it.

narrieta · 2022-10-07T20:17:46Z

azurelinuxagent/common/protocol/wire.py

@@ -604,35 +606,39 @@ def hgap_download(uri):

        return self._download_with_fallback_channel(download_type, uris, direct_download=direct_download, hgap_download=hgap_download)

-    def download_extension(self, uris, destination, use_verify_header, on_downloaded=lambda: True):


renamed this method since it can be used to download the agent as well; also, now this method can only be used to download ZIPs (we don't download any other type of files) and the expansion of the package is done here, instead of in the caller.

narrieta · 2022-10-07T20:18:37Z

azurelinuxagent/common/protocol/wire.py

+            raise
+        finally:
+            try:
+                os.remove(target_file)


now we remove the ZIP file after expansion

codecov · 2022-10-07T20:20:20Z

Codecov Report

Merging #2682 (0bf267a) into develop (faa8b14) will increase coverage by 0.03%.
The diff coverage is 96.29%.

@@             Coverage Diff             @@
##           develop    #2682      +/-   ##
===========================================
+ Coverage    72.02%   72.06%   +0.03%     
===========================================
  Files          103      103              
  Lines        15740    15702      -38     
  Branches      2254     2237      -17     
===========================================
- Hits         11337    11315      -22     
+ Misses        3881     3872       -9     
+ Partials       522      515       -7

Impacted Files	Coverage Δ
azurelinuxagent/common/protocol/goal_state.py	`94.76% <ø> (+0.19%)`	⬆️
azurelinuxagent/common/protocol/wire.py	`78.24% <93.10%> (+0.40%)`	⬆️
azurelinuxagent/ga/exthandlers.py	`85.67% <100.00%> (+0.02%)`	⬆️
azurelinuxagent/ga/update.py	`90.70% <100.00%> (+1.33%)`	⬆️
azurelinuxagent/common/cgroupapi.py	`81.00% <0.00%> (-2.80%)`	⬇️
azurelinuxagent/common/protocol/hostplugin.py	`87.76% <0.00%> (+0.53%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

narrieta · 2022-10-07T20:21:37Z

azurelinuxagent/ga/update.py

@@ -1494,18 +1488,20 @@ def _reset_legacy_blacklisted_agents(self):


 class GuestAgent(object):
-    def __init__(self, path=None, pkg=None, is_fast_track_goal_state=False, host=None):
+    def __init__(self, path, pkg, protocol, is_fast_track_goal_state):


The interface for init is confusing, since it can be used for 2 mutually exclusive purposes depending on the parameters given/omitted. I defined 'from_installed_agent' and 'from_installed_agent' that can be used instead of init with a (hopefully) clearer interface

Thanks for changing this

narrieta · 2022-10-07T20:23:12Z

azurelinuxagent/ga/update.py

@@ -1529,26 +1525,12 @@ def __init__(self, path=None, pkg=None, is_fast_track_goal_state=False, host=Non
            self._ensure_downloaded()
            self._ensure_loaded()
        except Exception as e:
-            if isinstance(e, ResourceGoneError):


ResourceGoneError is already handled in the common download logic, no need to treat it as a special case here.

narrieta · 2022-10-07T20:24:39Z

azurelinuxagent/ga/update.py

-            # encountered while downloading a later version. Errors of type
-            # socket.error are IOError, so this should provide sufficient
-            # protection against a large class of I/O operation failures.
-            if isinstance(e, IOError):


This check seems to be leftover code that is not needed. There is a unit test that verifies we do not blacklist the agent on download error so these errors are handled correctly even without this check.

narrieta · 2022-10-07T20:25:34Z

azurelinuxagent/ga/update.py

@@ -1637,39 +1632,10 @@ def _ensure_loaded(self):
        self._load_error()

    def _download(self):
-        uris_shuffled = self.pkg.uris


now we use the common download logic instead of this (as well as _fetch and _unpack below)

narrieta · 2022-10-07T20:27:17Z

tests/ga/test_extension.py

@@ -140,8 +141,6 @@ def test_cleanup_leaves_installed_extensions(self):
            exthandlers_handler.run()
            exthandlers_handler.report_ext_handlers_status()

-            self.assertEqual(no_of_exts, TestExtensionCleanup._count_packages(),


The ZIP package is now deleted after expansion so all these tests do not need to check the count of packages

tests/ga/test_extension.py

narrieta · 2022-10-07T20:31:00Z

tests/protocol/test_hostplugin.py

@@ -163,7 +163,7 @@ def create_mock_protocol():
            yield protocol

    @patch("azurelinuxagent.common.protocol.healthservice.HealthService.report_host_plugin_versions")
-    @patch("azurelinuxagent.ga.update.restutil.http_get")


This is sad. We were mocking restutil.http_get via the update module, which is not even used by this test.

yeah, I have seen these few places, thanks for correcting.

narrieta · 2022-10-07T20:32:53Z

tests/protocol/test_imds.py


 from azurelinuxagent.common.datacontract import set_properties
 from azurelinuxagent.common.exception import HttpError, ResourceGoneError
 from azurelinuxagent.common.future import ustr, httpclient
 from azurelinuxagent.common.utils import restutil
-from tests.ga.test_update import ResponseMock


There are multiple implementations of mock HTTP responses that make things confusing and cause some really weird test failures. Removing the usage of this one in favor of MockHttpResponse, which provides a more complete implementation

tests/ga/test_extension.py

nagworld9 · 2022-10-14T18:01:45Z

tests/protocol/test_hostplugin.py

@@ -163,7 +163,7 @@ def create_mock_protocol():
            yield protocol

    @patch("azurelinuxagent.common.protocol.healthservice.HealthService.report_host_plugin_versions")
-    @patch("azurelinuxagent.ga.update.restutil.http_get")


yeah, I have seen these few places, thanks for correcting.

nagworld9 · 2022-10-14T18:05:48Z

azurelinuxagent/ga/update.py

@@ -1494,18 +1488,20 @@ def _reset_legacy_blacklisted_agents(self):


 class GuestAgent(object):
-    def __init__(self, path=None, pkg=None, is_fast_track_goal_state=False, host=None):
+    def __init__(self, path, pkg, protocol, is_fast_track_goal_state):


Thanks for changing this

nagworld9 · 2022-10-14T18:45:17Z

azurelinuxagent/ga/update.py

            try:
                if os.path.isdir(self.get_agent_dir()):
                    shutil.rmtree(self.get_agent_dir(), ignore_errors=True)
-                if os.path.isfile(self.get_agent_pkg_path()):
-                    os.remove(self.get_agent_pkg_path())


I feel like we should keep this too which gives second chance to remove incase first attempt failed or didn't get to the unzip package path.

we still have the cleanup logic that runs periodically.

also, a new goal state would find that the ZIP is there and skip the download (and remove the ZIP after expanding it)

I don't find the cleanup logic for agent ZIPs. When I check the vm, zips still there. I don't think we have cleanup thing.

I'm more worry about if some reason agent gets partial zip or corrupted zip while downloading and received exception before expanding it. In that case on next goal state, we would skip the download if that zip is still in the vm.

A bad download should be covered as well.

The previous logic was trying to delete on download or expand failures. That cleanup is done covered elsewhere: if we begin the download, start creating the file and then downloading the rest of the file fails, the WireClient.stream would delete the ZIP. If the download succeeds, but the ZIP is corrupt and expand fails, then WireClient._try_expand_zip_package would remove it.

I was mistaken in my previous comment: a new goal state does not check for the existence of the ZIP, but rather for the manifest file. If for any reason a bad ZIP is still there from a previous goal state, the new goal state would do the download and overwrite the bad ZIP.

Lastly, during agent update, UpdateHandler._purge_agents removes agent directories and ZIPs that are no longer in the goal state.

All of this maintains the same logic as before, except that some of the resposibilities have moved elsewhere as part of using the common download logic.

Sonds like we are covering all the cases, thanks for explaining.

narrieta · 2022-10-07T20:36:37Z

tests/test_agent.py

            def mock_cgroup_paths(*args, **kwargs):
                if args and args[0] == "self":
                    relative_path = "{0}/{1}".format(cgroupconfigurator.LOGCOLLECTOR_SLICE, logcollector.CGROUPS_UNIT)
                    return (cgroupconfigurator.LOGCOLLECTOR_SLICE, relative_path)
                return SystemdCgroupsApi.get_process_cgroup_relative_paths(*args, **kwargs)

-            with patch.object(SystemdCgroupsApi, "get_process_cgroup_relative_paths", mock_cgroup_paths):


The mocks for the log collector tests need improvement. These 2 tests were failing on Ubuntu 22 because by default there no cgroup controllers mounted. Worked around the issue by moving the mock one level up the call stack.

narrieta · 2022-10-07T20:38:15Z

tests/ga/test_update.py

        self.assertFalse(agent.is_blacklisted)
        self.assertEqual(agent.is_blacklisted, agent.error.is_blacklisted)

        agent.mark_failure(is_fatal=True)
        self.assertTrue(agent.is_blacklisted)
        self.assertEqual(agent.is_blacklisted, agent.error.is_blacklisted)

-    @patch("azurelinuxagent.ga.update.GuestAgent._ensure_downloaded")
-    @patch("azurelinuxagent.ga.update.GuestAgent._ensure_loaded")
-    def test_resource_gone_error_not_blacklisted(self, mock_loaded, mock_downloaded):  # pylint: disable=unused-argument


ResourceGone error is handled by the common download logic and GuestAgent does not handle it anymore so this test is not needed

narrieta · 2022-10-07T20:38:59Z

tests/ga/test_update.py

-
-    @patch("azurelinuxagent.ga.update.GuestAgent._ensure_downloaded")
-    @patch("azurelinuxagent.ga.update.GuestAgent._ensure_loaded")
-    def test_ioerror_not_blacklisted(self, mock_loaded, mock_downloaded):  # pylint: disable=unused-argument


This was testing a left over behavior that I removed. Removing as well.

'test_download_fail' checks that the agent is not blacklisted on download error

narrieta · 2022-10-07T20:43:17Z

tests/ga/test_update.py

@@ -521,225 +485,115 @@ def test_mark_failure(self, mock_loaded, mock_downloaded):  # pylint: disable=un
        self.assertEqual(2, agent.error.failure_count)
        self.assertTrue(agent.is_blacklisted)

-    @patch("azurelinuxagent.ga.update.GuestAgent._ensure_downloaded")
-    @patch("azurelinuxagent.ga.update.GuestAgent._ensure_loaded")
-    def test_unpack(self, mock_loaded, mock_downloaded):  # pylint: disable=unused-argument


unpack is done in the common logic, the test is no longer needed

narrieta · 2022-10-07T20:45:12Z

tests/ga/test_update.py

        self.assertFalse(agent.is_blacklisted)
        self.assertEqual(agent.is_blacklisted, agent.error.is_blacklisted)

        agent.mark_failure(is_fatal=True)
        self.assertTrue(agent.is_blacklisted)
        self.assertEqual(agent.is_blacklisted, agent.error.is_blacklisted)

-    @patch("azurelinuxagent.ga.update.GuestAgent._ensure_downloaded")


many of those tests were mocking stuff that is not even executed by the test, seems like copy-and-paste implementation. the changes below remove those unneeded mocks

narrieta · 2022-10-07T20:46:20Z

tests/ga/test_update.py

-    @patch("azurelinuxagent.ga.update.GuestAgent._ensure_loaded")
-    @patch("azurelinuxagent.ga.update.restutil.http_get")
-    @patch("azurelinuxagent.ga.update.restutil.http_post")
-    def test_download_fallback(self, mock_http_post, mock_http_get, mock_loaded, mock_downloaded):  # pylint: disable=unused-argument


this was testing the logic that was removed to used the common download implementation

narrieta · 2022-10-07T20:46:58Z

tests/ga/test_update.py

@@ -990,15 +844,6 @@ def test_find_agents_sorts(self):
            self.assertTrue(v > a.version)
            v = a.version

-    @patch('azurelinuxagent.common.protocol.wire.WireClient.get_host_plugin')
-    def test_get_host_plugin_returns_host_for_wireserver(self, mock_get_host):


The tested method was removed. Removing test

narrieta · 2022-10-17T22:17:19Z

azurelinuxagent/ga/update.py

            try:
                if os.path.isdir(self.get_agent_dir()):
                    shutil.rmtree(self.get_agent_dir(), ignore_errors=True)
-                if os.path.isfile(self.get_agent_pkg_path()):
-                    os.remove(self.get_agent_pkg_path())


we still have the cleanup logic that runs periodically.

also, a new goal state would find that the ZIP is there and skip the download (and remove the ZIP after expanding it)

tests/ga/test_extension.py

narrieta added 2 commits October 6, 2022 12:12

Use common download logic for agent downloads

1050e70

rename method

97ddcf7

narrieta requested review from ZhidongPeng, nagworld9 and maddieford as code owners October 7, 2022 20:10

Merge branch 'develop' into agent-download

107cbfd

narrieta commented Oct 7, 2022

View reviewed changes

tests/ga/test_extension.py Show resolved Hide resolved

narrieta commented Oct 7, 2022

View reviewed changes

nagworld9 reviewed Oct 14, 2022

View reviewed changes

narrieta commented Oct 17, 2022

View reviewed changes

add unit test

0bf267a

nagworld9 approved these changes Oct 18, 2022

View reviewed changes

narrieta merged commit 527443c into Azure:develop Oct 18, 2022

narrieta deleted the agent-download branch October 18, 2022 16:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use common download logic for agent downloads #2682

Use common download logic for agent downloads #2682

narrieta commented Oct 7, 2022 •

edited

Loading

narrieta Oct 7, 2022

narrieta Oct 7, 2022

narrieta Oct 7, 2022 •

edited

Loading

codecov bot commented Oct 7, 2022 •

edited

Loading

narrieta Oct 7, 2022

nagworld9 Oct 14, 2022

narrieta Oct 7, 2022

narrieta Oct 7, 2022

narrieta Oct 7, 2022 •

edited

Loading

narrieta Oct 7, 2022

narrieta Oct 7, 2022

nagworld9 Oct 14, 2022

narrieta Oct 7, 2022

nagworld9 Oct 14, 2022

nagworld9 Oct 14, 2022

nagworld9 Oct 14, 2022

narrieta Oct 17, 2022

nagworld9 Oct 18, 2022

narrieta Oct 18, 2022

nagworld9 Oct 18, 2022

narrieta Oct 7, 2022

narrieta Oct 7, 2022

narrieta Oct 7, 2022

narrieta Oct 7, 2022

narrieta Oct 7, 2022

narrieta Oct 7, 2022

narrieta Oct 7, 2022

narrieta Oct 17, 2022

		@@ -604,35 +606,39 @@ def hgap_download(uri):

		return self._download_with_fallback_channel(download_type, uris, direct_download=direct_download, hgap_download=hgap_download)

		def download_extension(self, uris, destination, use_verify_header, on_downloaded=lambda: True):

Use common download logic for agent downloads #2682

Use common download logic for agent downloads #2682

Conversation

narrieta commented Oct 7, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

narrieta Oct 7, 2022 • edited Loading

Choose a reason for hiding this comment

codecov bot commented Oct 7, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

narrieta Oct 7, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

narrieta commented Oct 7, 2022 •

edited

Loading

narrieta Oct 7, 2022 •

edited

Loading

codecov bot commented Oct 7, 2022 •

edited

Loading

narrieta Oct 7, 2022 •

edited

Loading