[2.2.14] HostGAPlugin still use proxy while auto-updating

[yuxisun1217]

Hi there,

I saw there was an issue #769 "HostGAPlugin requests should never go through proxy". But currently it seems that only the blob status upload traffic can ignore the proxy when proxy is misconfigured. For the auto-update feature, only the manifest file can be downloaded without proxy, the WALA update packages still use proxy and fail to be downloaded. The extension downloading process is the same as auto-update. Is it by design? Thanks!

My steps:

1. Configure /etc/waagent.conf to set a wrong proxy, enable auto-update, and enable verbose log:
   HttpProxy.Host=172.16.0.1
   HttpProxy.Port=3128
   Logs.Verbose=y
   AutoUpdate.Enabled=y
   Service waagent restart
2. Check /var/log/waagent.log

2017/07/12 06:29:39.476351 VERBOSE HTTP proxy: [172.16.0.1:3128]
2017/07/12 06:29:39.483259 VERBOSE HTTP connection [GET] [http://rdfepirv2sg1prdstr03.blob.core.windows.net:80/7d89d439b79f4452950452399add2c90/Microsoft.OSTCLinuxAgent_Prod_asiasoutheast_manifest.xml] [None] [None]
2017/07/12 06:29:49.513148 VERBOSE HTTP connection [GET] [http://rdfepirv2sg1prdstr03.blob.core.windows.net:80/7d89d439b79f4452950452399add2c90/Microsoft.OSTCLinuxAgent_Prod_asiasoutheast_manifest.xml] [None] [None]
2017/07/12 06:29:59.547066 VERBOSE HTTP connection [GET] [http://rdfepirv2sg1prdstr03.blob.core.windows.net:80/7d89d439b79f4452950452399add2c90/Microsoft.OSTCLinuxAgent_Prod_asiasoutheast_manifest.xml] [None] [None]
2017/07/12 06:30:09.572601 VERBOSE Fetch failed from [https://rdfepirv2sg1prdstr03.blob.core.windows.net/7d89d439b79f4452950452399add2c90/Microsoft.OSTCLinuxAgent_Prod_asiasoutheast_manifest.xml]: [000009] HTTP GET failed
2017/07/12 06:30:09.591745 VERBOSE Manifest could not be downloaded, falling back to host plugin
2017/07/12 06:30:09.602838 VERBOSE HostGAPlugin: Getting API versions at [http://100.107.240.13:32526/versions]
2017/07/12 06:30:09.614843 VERBOSE HTTP connection [GET] [/versions] [None] [{'x-ms-containerid': u'd4384bb2-4f5c-4680-8c3d-9fa51841ba7d'}]
2017/07/12 06:30:09.632135 VERBOSE HTTP response status: [200]
2017/07/12 06:30:09.639179 INFO Event: name=WALinuxAgent, op=InitializeHostPlugin, message=
2017/07/12 06:30:09.650182 VERBOSE Fetch [http://100.107.240.13:32526/extensionArtifact] with headers [{'x-ms-containerid': u'd4384bb2-4f5c-4680-8c3d-9fa51841ba7d', 'x-ms-version': '2015-09-01', 'x-ms-host-config-name': u'77bbebc9f3994cc48fbef61834a1822e.0.77bbebc9f3994cc48fbef61834a1822e.0.wala692214ui4.1.xml', 'x-ms-artifact-location': u'https://rdfepirv2sg1prdstr03.blob.core.windows.net/7d89d439b79f4452950452399add2c90/Microsoft.OSTCLinuxAgent_Prod_asiasoutheast_manifest.xml'}]
2017/07/12 06:30:09.689246 VERBOSE HTTP connection [GET] [/extensionArtifact] [None] [{'x-ms-containerid': u'd4384bb2-4f5c-4680-8c3d-9fa51841ba7d', 'x-ms-version': '2015-09-01', 'x-ms-host-config-name': u'77bbebc9f3994cc48fbef61834a1822e.0.77bbebc9f3994cc48fbef61834a1822e.0.wala692214ui4.1.xml', 'x-ms-artifact-location': u'https://rdfepirv2sg1prdstr03.blob.core.windows.net/7d89d439b79f4452950452399add2c90/Microsoft.OSTCLinuxAgent_Prod_asiasoutheast_manifest.xml'}]
2017/07/12 06:30:09.763930 VERBOSE HTTP response status: [200]
2017/07/12 06:30:09.770335 VERBOSE Manifest downloaded successfully from host plugin
2017/07/12 06:30:09.778017 INFO Setting host plugin as default channel
2017/07/12 06:30:09.785914 VERBOSE Load ExtensionManifest.xml
2017/07/12 06:30:09.794175 VERBOSE Loading Agent WALinuxAgent-2.2.10 from package package
2017/07/12 06:30:09.803674 VERBOSE Agent WALinuxAgent-2.2.10 error state: Last Failure: 0.0, Total Failures: 0, Fatal: False
2017/07/12 06:30:09.816401 VERBOSE Ensuring Agent WALinuxAgent-2.2.10 is downloaded
2017/07/12 06:30:09.825242 VERBOSE Using host plugin as default channel
2017/07/12 06:30:09.833194 VERBOSE HTTP proxy: [172.16.0.1:3128]
2017/07/12 06:30:09.839940 VERBOSE HTTP connection [GET] [http://100.107.240.13:32526/extensionArtifact] [None] [{'x-ms-containerid': u'd4384bb2-4f5c-4680-8c3d-9fa51841ba7d', 'x-ms-artifact-manifest-location': u'https://rdfepirv2sg1prdstr03.blob.core.windows.net/7d89d439b79f4452950452399add2c90/Microsoft.OSTCLinuxAgent_Prod_asiasoutheast_manifest.xml', 'x-ms-version': '2015-09-01', 'x-ms-host-config-name': u'77bbebc9f3994cc48fbef61834a1822e.0.77bbebc9f3994cc48fbef61834a1822e.0.wala692214ui4.1.xml', 'x-ms-artifact-location': u'https://rdfepirv2sg1prdstr03.blob.core.windows.net/7d89d439b79f4452950452399add2c90/Microsoft.OSTCLinuxAgent__Prod__2.2.10'}]
2017/07/12 06:30:19.911177 VERBOSE HTTP connection [GET] [http://100.107.240.13:32526/extensionArtifact] [None] [{'x-ms-containerid': u'd4384bb2-4f5c-4680-8c3d-9fa51841ba7d', 'x-ms-artifact-manifest-location': u'https://rdfepirv2sg1prdstr03.blob.core.windows.net/7d89d439b79f4452950452399add2c90/Microsoft.OSTCLinuxAgent_Prod_asiasoutheast_manifest.xml', 'x-ms-version': '2015-09-01', 'x-ms-host-config-name': u'77bbebc9f3994cc48fbef61834a1822e.0.77bbebc9f3994cc48fbef61834a1822e.0.wala692214ui4.1.xml', 'x-ms-artifact-location': u'https://rdfepirv2sg1prdstr03.blob.core.windows.net/7d89d439b79f4452950452399add2c90/Microsoft.OSTCLinuxAgent__Prod__2.2.10'}]
2017/07/12 06:30:29.992187 VERBOSE HTTP connection [GET] [http://100.107.240.13:32526/extensionArtifact] [None] [{'x-ms-containerid': u'd4384bb2-4f5c-4680-8c3d-9fa51841ba7d', 'x-ms-artifact-manifest-location': u'https://rdfepirv2sg1prdstr03.blob.core.windows.net/7d89d439b79f4452950452399add2c90/Microsoft.OSTCLinuxAgent_Prod_asiasoutheast_manifest.xml', 'x-ms-version': '2015-09-01', 'x-ms-host-config-name': u'77bbebc9f3994cc48fbef61834a1822e.0.77bbebc9f3994cc48fbef61834a1822e.0.wala692214ui4.1.xml', 'x-ms-artifact-location': u'https://rdfepirv2sg1prdstr03.blob.core.windows.net/7d89d439b79f4452950452399add2c90/Microsoft.OSTCLinuxAgent__Prod__2.2.10'}]
2017/07/12 06:30:39.543400 VERBOSE Found event file: /var/lib/waagent/events/1499855409649965.tld
2017/07/12 06:30:39.554873 VERBOSE Processed event file: /var/lib/waagent/events/1499855409649965.tld
2017/07/12 06:30:39.566265 VERBOSE HTTP connection [POST] [/machine?comp=telemetrydata] [<?xml version="1.0"?><TelemetryData version="1.0"><Provider id="69B669B9-4AF8-4C50-BDC4-6006FA76E975"><Event id="1"><![CDATA[<Param Name="Name" Value="WALinuxAgent" T="mt:wstr" /><Param Name="Version" Value="2.2.14" T="mt:wstr" /><Param Name="IsInternal" Value="False" T="mt:bool" /><Param Name="Operation" Value="InitializeHostPlugin" T="mt:wstr" /><Param Name="OperationSuccess" Value="True" T="mt:bool" /><Param Name="Message" Value="" T="mt:wstr" /><Param Name="Duration" Value="0" T="mt:uint64" /><Param Name="ExtensionType" Value="" T="mt:wstr" /><Param Name="OSVersion" Value="Linux:redhat-6.9-Santiago:2.6.32-696.3.2.el6.x86_64" T="mt:wstr" /><Param Name="GAVersion" Value="WALinuxAgent-2.2.14" T="mt:wstr" /><Param Name="RAM" Value="1679" T="mt:uint64" /><Param Name="Processors" Value="1" T="mt:uint64" /><Param Name="VMName" Value="wala692214ui4" T="mt:wstr" /><Param Name="TenantName" Value="77bbebc9f3994cc48fbef61834a1822e" T="mt:wstr" /><Param Name="RoleName" Value="wala692214ui4" T="mt:wstr" /><Param Name="RoleInstanceName" Value="77bbebc9f3994cc48fbef61834a1822e.wala692214ui4" T="mt:wstr" /><Param Name="ContainerId" Value="d4384bb2-4f5c-4680-8c3d-9fa51841ba7d" T="mt:wstr" />]]></Event></Provider></TelemetryData>] [{'Content-Type': 'text/xml;charset=utf-8', 'x-ms-version': '2012-11-30', 'x-ms-agent-name': 'WALinuxAgent'}]
2017/07/12 06:30:39.768153 VERBOSE HTTP response status: [200]
2017/07/12 06:30:40.064471 VERBOSE Agent WALinuxAgent-2.2.10 download from http://100.107.240.13:32526/extensionArtifact failed [[000009] HTTP GET failed]
2017/07/12 06:30:40.079068 WARNING Host plugin download unsuccessful
2017/07/12 06:30:40.089613 VERBOSE Using host plugin as default channel
2017/07/12 06:30:40.097071 VERBOSE HTTP proxy: [172.16.0.1:3128]
2017/07/12 06:30:40.103549 VERBOSE HTTP connection [GET] [http://100.107.240.13:32526/extensionArtifact] [None] [{'x-ms-containerid': u'd4384bb2-4f5c-4680-8c3d-9fa51841ba7d', 'x-ms-artifact-manifest-location': u'https://rdfepirv2sg1prdstr03.blob.core.windows.net/7d89d439b79f4452950452399add2c90/Microsoft.OSTCLinuxAgent_Prod_asiasoutheast_manifest.xml', 'x-ms-version': '2015-09-01', 'x-ms-host-config-name': u'77bbebc9f3994cc48fbef61834a1822e.0.77bbebc9f3994cc48fbef61834a1822e.0.wala692214ui4.1.xml', 'x-ms-artifact-location': u'https://rdfepirv2sg1prdstr04.blob.core.windows.net/7d89d439b79f4452950452399add2c90/Microsoft.OSTCLinuxAgent__Prod__2.2.10'}]
2017/07/12 06:30:50.176833 VERBOSE HTTP connection [GET] [http://100.107.240.13:32526/extensionArtifact] [None] [{'x-ms-containerid': u'd4384bb2-4f5c-4680-8c3d-9fa51841ba7d', 'x-ms-artifact-manifest-location': u'https://rdfepirv2sg1prdstr03.blob.core.windows.net/7d89d439b79f4452950452399add2c90/Microsoft.OSTCLinuxAgent_Prod_asiasoutheast_manifest.xml', 'x-ms-version': '2015-09-01', 'x-ms-host-config-name': u'77bbebc9f3994cc48fbef61834a1822e.0.77bbebc9f3994cc48fbef61834a1822e.0.wala692214ui4.1.xml', 'x-ms-artifact-location': u'https://rdfepirv2sg1prdstr04.blob.core.windows.net/7d89d439b79f4452950452399add2c90/Microsoft.OSTCLinuxAgent__Prod__2.2.10'}]
2017/07/12 06:31:00.249870 VERBOSE HTTP connection [GET] [http://100.107.240.13:32526/extensionArtifact] [None] [{'x-ms-containerid': u'd4384bb2-4f5c-4680-8c3d-9fa51841ba7d', 'x-ms-artifact-manifest-location': u'https://rdfepirv2sg1prdstr03.blob.core.windows.net/7d89d439b79f4452950452399add2c90/Microsoft.OSTCLinuxAgent_Prod_asiasoutheast_manifest.xml', 'x-ms-version': '2015-09-01', 'x-ms-host-config-name': u'77bbebc9f3994cc48fbef61834a1822e.0.77bbebc9f3994cc48fbef61834a1822e.0.wala692214ui4.1.xml', 'x-ms-artifact-location': u'https://rdfepirv2sg1prdstr04.blob.core.windows.net/7d89d439b79f4452950452399add2c90/Microsoft.OSTCLinuxAgent__Prod__2.2.10'}]
2017/07/12 06:31:10.331413 VERBOSE Agent WALinuxAgent-2.2.10 download from http://100.107.240.13:32526/extensionArtifact failed [[000009] HTTP GET failed]
2017/07/12 06:31:10.347366 WARNING Host plugin download unsuccessful
2017/07/12 06:31:10.355276 VERBOSE Using host plugin as default channel
2017/07/12 06:31:10.363528 VERBOSE HTTP proxy: [172.16.0.1:3128]
...

[brendandixon]

@yuxisun1217 Good find! Thank you.

[brendandixon]

Addressed by #814

[yuxisun1217]

Hi @brendandixon ,

The Extension downloading part still has this issue in WALA-2.2.18. Maybe need to set use_proxy=False in wire.py and restapi.py. Such as:

restapi.py:

324     def download_ext_handler_pkg(self, uri, headers=None, use_proxy=True):
325         try:
326             resp = restutil.http_get(uri, use_proxy=use_proxy, headers=headers)

wire.py:

 166     def download_ext_handler_pkg(self, uri, headers=None):
...
 171         else:
 172             logger.warn("Download did not succeed, falling back to host plugin")
 173             host = self.client.get_host_plugin()
 174             uri, headers = host.get_artifact_request(uri, host.manifest_uri)
 175             package = super(WireProtocol, self).download_ext_handler_pkg(uri, headers=headers, use_proxy=False)
 176         return package

[brendandixon]

@yuxisun1217 In reviewing this, I believe we do have an issue here.