-
Notifications
You must be signed in to change notification settings - Fork 371
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OS.EnableFirewall=y breaks load balanced sets probing... #879
Comments
@rkirkpat We apologize that this caused you time and pain. The Azure fabric IP address is not meant for general consumption. Restricting access to the Linux Agent process (which is, effectively, what the What probe are your services dependent upon? I'm unaware of any probe from the Azure fabric into a service. |
Hi, we experienced the same issue. After the WALinuxAgent update our web services and the web app were no longer available through the load balancer.
We are not using Azure Fabric, the WALinuxAgent was installed by default. |
Signed-off-by: Brendan Dixon <brendandixon@me.com>
* [#879] -- OS.EnableFirewall=y breaks load balanced sets probing Signed-off-by: Brendan Dixon <brendandixon@me.com> * Set version to v2.2.17 Signed-off-by: Brendan Dixon <brendandixon@me.com>
Address by #883 |
* [#879] -- OS.EnableFirewall=y breaks load balanced sets probing Signed-off-by: Brendan Dixon <brendandixon@me.com> * Set version to v2.2.17 Signed-off-by: Brendan Dixon <brendandixon@me.com>
Not sure why it was closed, I've just deployed VM and in /etc/waagent.conf And still: https://github.com/Azure/WALinuxAgent/blob/master/config/ubuntu/waagent.conf#L107 The comment says that by default it should be |
@Suvitruf - the initial firewall rule was disabled because it was too restrictive, and hence this issue was closed. Since then we have started rolling out essentially the same functional change but with a less restrictive rule, which should not affect load balancer probes. Thanks for pointing out the comment in the config needs to be updated, I have opened #1260 for that. |
We have several Ubuntu 14.04 LTS (classic) VMs in the Azure cloud running HTTPS web services on port 443. These web services are exposed to the Internet using load balanced sets with the probe port set also to be 443. Yesterday we upgraded these VMs with security updates, including an update of
walinuxagent
from v2.0.14 to v2.0.16, after which these web services were no longer accessible.After much troubleshooting we discovered that the probes sent from Azure fabric IP, 168.63.129.16, were never getting a reply from our servers, as per this
tcpdump
output:We then proceeded to revert the updated packages one by one and eventually found that the updated
walinuxagent
package was the cause of failure. Reviewing/etc/waagent.conf
we found a new config options,OS.EnableFirewall
, and that it was enabled. Once we disabled that option and rebooted the server (on one that had not been downgraded), the web services were accessible again as the probe requests were getting responses now:We reviewed the commits to the
waagent.conf
file on GitHub and found that a recent commit, e247e7b, had added this option and firewall rules blocking any non-root process from communicating with the fabric server 168.63.129.16. Of course our web services on port 443 are not running as root (it is a custom twisted python service running as a service user) and hence are not allowed to receive the probe from the fabric.There was no warning about this change in any release notes, and it was enabled by default (in conflict with the comment directly above it in the config file that by default it was to be disabled). This issue cost us quite a bit of engineering time to find the solution and restore our web services. I would recommend this option be disabled by default or at least the user warned about it being enabled!
The text was updated successfully, but these errors were encountered: