(000008)Failed to fetch ExtensionManifest from all sources #458
Comments
|
@sweekarp Usually network issues, such as overly restrictive VNets (e.g., disallowing public IP access), cause these problems. In a few rare cases, the extension is no longer available (which, naturally raises the question of why the agent was asked to install it...). We've not had sufficient data on these latter cases to run them down. I do know that the agent is never at fault. What extension is failing to install? The logs also normally contain the failing URI. Have you tried to access that URI directly (e.g., in a browser). If it works, which is normally the case, it means the VNet is not allowing the outbound access needed to install the extension. |
|
@brendandixon Trying to access the URI directly results in bad request: curl http://10.94.194.11/machine/?comp=goalstate Full verbose log attached |
|
@sweekarp The attached logs show the agent successfully reaching wireserver (the service that handles the request above). You cannot use curl to access this endpoint as it requires HTTP headers. If you want to see the files retrieved, the agent stores them in /var/lib/waagent. I would have to see the Xml files the agent received to determine why the ExtensionManifest failed to download. It is likely a transient problem. |
|
@yuxisun1217 Thank you! I found the bug. It's not serious: The error message is noise and the agent continues run and "do the right thing." Still, it should not emit the message. |
|
@yuxisun1217 @sweekarp #462 corrects the problem of attempting to retrieve missing URIs. |
|
Thanks @brendandixon ! By the way, could you please help to explain that why there is no URI in the Prod family? Does it mean that currently there's no WALA package on Azure to self-update? |
|
@yuxisun1217 We tag agent updates by "family" (e.g., Test, Prod). The lack of URIs in the document you provided means that no updates within the "Prod" family exist. Since we do not release updates simultaneously across all regions, to better test them with production workloads, it is not uncommon to see this. |
|
@brendandixon Got it. Thank you very much! |
|
Closing the issue as #462 fixes the issue. |
I get the following error message when running WALinuxAgent-2.2.0 on an Oracle Linux 6 or 7 guest on Azure:
Here's the full log:
2016/10/05 17:01:58.005301 INFO Agent WALinuxAgent-2.2.0 forwarding signal 15 to WALinuxAgent-2.2.0
2016/10/05 17:01:58.465810 INFO Azure Linux Agent Version:2.2.0
2016/10/05 17:01:58.478311 INFO OS: oracle 7.2
2016/10/05 17:01:58.483163 INFO Python: 2.7.5
2016/10/05 17:01:58.488343 INFO Run daemon
2016/10/05 17:01:58.493861 INFO No RDMA handler exists for distro='Oracle Linux' version='7.2'
2016/10/05 17:01:58.502284 INFO Activate resource disk
2016/10/05 17:01:58.523614 INFO Resource disk [/dev/sdb1] is already mounted [/mnt/resource]
2016/10/05 17:01:58.539728 INFO Clean protocol
2016/10/05 17:01:58.545434 INFO Provisioning already completed, skipping.
2016/10/05 17:01:58.553491 INFO RDMA capabilities are not enabled, skipping
2016/10/05 17:01:58.561690 INFO Instantiating Agent WALinuxAgent-2.1.6 from disk
2016/10/05 17:01:58.569900 INFO Agent WALinuxAgent-2.1.6 error state: Last Failure: 0.0, Total Failures: 0, Fatal: False
2016/10/05 17:01:58.582779 INFO Ensuring Agent WALinuxAgent-2.1.6 is downloaded
2016/10/05 17:01:58.590603 INFO Agent WALinuxAgent-2.1.6 was previously downloaded - skipping download
2016/10/05 17:01:58.600538 INFO Agent WALinuxAgent-2.1.6 loaded manifest from /var/lib/waagent/WALinuxAgent-2.1.6/HandlerManifest.json
2016/10/05 17:01:58.613295 INFO Installed Agent WALinuxAgent-2.2.0 is the most current agent
2016/10/05 17:01:58.632902 INFO Agent WALinuxAgent-2.2.0 launched with command 'python -u /usr/sbin/waagent -run-exthandlers'
2016/10/05 17:01:58.788546 INFO Agent WALinuxAgent-2.2.0 is running as the goal state agent
2016/10/05 17:01:58.798161 INFO Detect protocol endpoints
2016/10/05 17:01:58.804753 INFO Clean protocol
2016/10/05 17:01:58.810079 INFO WireServer endpoint is not found. Rerun dhcp handler
2016/10/05 17:01:58.818996 INFO test for route to 168.63.129.16
2016/10/05 17:01:58.826539 WARNING no route exists to 168.63.129.16
2016/10/05 17:01:58.833753 INFO checking for dhcp lease cache
2016/10/05 17:01:58.840328 INFO looking for leases in path [/var/lib/dhclient/dhclient-*.lease]
2016/10/05 17:01:58.854037 INFO dhcp entry:10.92.44.9, 245:True, expired:False
2016/10/05 17:01:58.862158 INFO found endpoint [10.92.44.9]
2016/10/05 17:01:58.868304 INFO dhcp entry:10.92.52.17, 245:True, expired:False
2016/10/05 17:01:58.875952 INFO found endpoint [10.92.52.17]
2016/10/05 17:01:58.882358 INFO dhcp entry:10.92.52.17, 245:True, expired:False
2016/10/05 17:01:58.890165 INFO found endpoint [10.92.52.17]
2016/10/05 17:01:58.896336 INFO cached endpoint found [10.92.52.17]
2016/10/05 17:01:58.903131 INFO cache exists [True]
2016/10/05 17:01:58.908368 INFO Wire server endpoint:10.92.52.17
2016/10/05 17:01:58.918520 INFO Fabric preferred wire protocol version:2015-04-05
2016/10/05 17:01:58.926897 INFO Wire protocol version:2012-11-30
2016/10/05 17:01:58.933233 WARNING Server prefered version:2015-04-05
2016/10/05 17:02:02.829769 INFO Event: name=WALinuxAgent-2.2.0, op=HeartBeat, message=
2016/10/05 17:02:02.840533 INFO Start env monitor service.
2016/10/05 17:02:02.846878 INFO Configure routes
2016/10/05 17:02:02.852817 INFO Gateway:None
2016/10/05 17:02:02.858182 INFO Routes:None
2016/10/05 17:02:02.888089 INFO WALinuxAgent-2.2.0 running as process 32162
2016/10/05 17:02:02.902134 INFO Checking for agent family Prod updates
2016/10/05 17:02:02.909943 INFO Wire server endpoint:10.92.52.17
2016/10/05 17:02:02.923016 WARNING Incarnation 1 failed to get Prod package list: (000008)Failed to fetch ExtensionManifest from all sources
2016/10/05 17:02:02.941499 ERROR Event: name=WALinuxAgent, op=Download, message=Incarnation 1 failed to get Prod package list: (000008)Failed to fetch ExtensionManifest from all sources
2016/10/05 17:02:02.958846 INFO Wire server endpoint:10.92.52.17
2016/10/05 17:16:59.432130 INFO Agent WALinuxAgent-2.2.0 launched with command 'python -u /usr/sbin/waagent -run-exthandlers' is successfully running
2016/10/05 17:16:59.446300 INFO Event: name=WALinuxAgent, op=Enable, message=Agent WALinuxAgent-2.2.0 launched with command 'python -u /usr/sbin/waagent -run-exthandlers' is successfully running
2016/10/05 17:32:05.552672 INFO Event: name=WALinuxAgent-2.2.0, op=HeartBeat, message=
2016/10/05 18:02:08.188689 INFO Event: name=WALinuxAgent-2.2.0, op=HeartBeat, message=
2016/10/05 18:02:20.321522 INFO Checking for agent family Prod updates
2016/10/05 18:02:20.332534 WARNING Incarnation 1 failed to get Prod package list: (000008)Failed to fetch ExtensionManifest from all sources
2016/10/05 18:02:20.347000 ERROR Event: name=WALinuxAgent, op=Download, message=Incarnation 1 failed to get Prod package list: (000008)Failed to fetch ExtensionManifest from all sources
2016/10/05 18:32:09.829240 INFO Event: name=WALinuxAgent-2.2.0, op=HeartBeat, message=
2016/10/05 19:02:12.432196 INFO Event: name=WALinuxAgent-2.2.0, op=HeartBeat, message=
2016/10/05 19:02:35.372049 INFO Checking for agent family Prod updates
2016/10/05 19:02:36.384095 WARNING Incarnation 1 failed to get Prod package list: (000008)Failed to fetch ExtensionManifest from all sources
2016/10/05 19:02:36.397556 ERROR Event: name=WALinuxAgent, op=Download, message=Incarnation 1 failed to get Prod package list: (000008)Failed to fetch ExtensionManifest from all sources
The text was updated successfully, but these errors were encountered: