-
Notifications
You must be signed in to change notification settings - Fork 9.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Automated monitoring of an etcd cluster? #2383
Comments
@Telmo Thanks for reaching out. I'm going to propose a simple option, please let me know if this is what you are looking for.
We have a new health endpoint which you can test with the following command:
We have not documented the raw api call for this, but it might provided a better approach for monitoring the health of the entire cluster. Also, how should this work? Would you like a single endpoint that you can hit with a nagios check? |
a single endpoint would be definitely a plus, however I am aware that if a node drops from the cluster the cluster may not be aware that the node is not there any longer. What's the http endpoint for Basically, we are a BIG company, and we need to make sure our etcd cluster can be monitored so we can be proactive/reactive with regards of nodes being down or cluster issues. |
@Telmo I'll produce a monitoring guide early next week for your review. |
@Telmo I did not forget about you. I'm setting up our etcd integration environment now and I'm using Datadog for metrics and monitoring. Once I've got my datadog integration done I'll share it with you. |
Perfect, thank you very much for this. |
@kelseyhightower any updates? Looking forward to the guide. |
+1 |
1 similar comment
+1 |
@Telmo there is a plugin (or two) for sensu that help with this. sometimes /health goes false and then returns true 2s later; so i'm nervous of the cluster health monitoring although I want it. (Just don't want the false positives alerting) https://github.com/sensu/sensu-community-plugins/tree/master/plugins/etcd Being able to monitor if etcd is up; is fairly critical. Having metrics is useful; a check for /health would be good but it does return false occasionally so it'll have to have a lot of retries. |
Now etcd supports metrics reporting: https://github.com/coreos/etcd/blob/master/Documentation/metrics.md You can curl the /health endpoint of each member to get the health information. Thanks. |
cluster-health only shows health of members not proxies it would be nice to check if proxies were also up |
@hmatinho Proxy is not part of the actual etcd cluster. And you might have a large number of proxies. I do not think it is the responsibility of the etcd cluster to report the status of proxies. |
I am having a similar issue. Using F.e: I want to use the gce network loadbalancer with url based health checks. But it is not possible to provide a ca, cert or key. I have only one idea at the moment. Setup a nginx as proxy for a special route on a different port that just gives you the health back without tls. Does somebody has a better idea? It would be cool if you could enable a second http that has restricted access and serves without tls. |
Be aware that |
I've read all the tools using etcd, I've google different permutation of "etcd monitoring" and I am still unable to find a way to monitor an etcd cluster. With monitoring I mean a programatic way of checking the global health of a cluster, to be alerted if "nodes fall off the map" or etcd stops working all together.
Is there such a tool? If not what are people using to be alerted of possible issues with etcd?
The text was updated successfully, but these errors were encountered: