Q: Excessive “consul: Health changed to #XXX” messages in fabio's log

[hynek]

Hi,

it feels awkward to ask here but it seems to be the right place…

I’ve got Fabio up and running without any issues except, that I’m getting very frequent log entries like this:

Dec 12 18:39:41 c-b4fa fabio[25680]: 2016/12/12 18:39:41 [INFO] consul: Health changed to #54416876
Dec 12 18:39:41 c-b4fa fabio[25680]: 2016/12/12 18:39:41 [INFO] consul: Health changed to #54416877
Dec 12 18:39:43 c-b4fa fabio[25680]: 2016/12/12 18:39:43 [INFO] consul: Health changed to #54416882
Dec 12 18:39:43 c-b4fa fabio[25680]: 2016/12/12 18:39:43 [INFO] consul: Health changed to #54416883
Dec 12 18:39:44 c-b4fa fabio[25680]: 2016/12/12 18:39:44 [INFO] consul: Health changed to #54416884
Dec 12 18:39:48 c-b4fa fabio[25680]: 2016/12/12 18:39:48 [INFO] consul: Health changed to #54416894

They usually come at least once per second. Is that normal or is something weird about our Consul/Vault/Nomad/Fabio setup?

TIA, Fabio’s great!

[magiconair]

Check if one or more of your services have flapping or lagging health checks.

[magiconair]

@hynek As for the awkwardness see my comment on #203. Please feel free to ask at any time.

[magiconair]

@hynek I'm going to close this issue for now. Feel free to comment and I'll re-open it if necessary.

[hynek]

yes thank you, I’m still investigating :-/

nothing is flapping but i'm not sure what a lagging health check would be? in any case this is probably not a fabio issue…

happy holidays anyways. :)

[jorgemarey]

Hi, I've seen this exact same behaviour on my cluster. Just to throw some ideas, I've been debugging and found that is nomad the one causing this.

The check that is performed on nomad is on /v1/agent/servers which outputs the servers of the cluster.

The problem is that this array is not ordered and changes randomly (it's not random actually, but it is not guaranteed to be the same) on each request, so when there's more than one the check becomes the one causing the blocking query to end.

I'll open an issue on nomad to see what they think about this.

[magiconair]

@jorgemarey I'll ping them as well. Thx.

[dadgar]

This PR should fix the issue: hashicorp/nomad#3214