Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Health checks, Kubernetes probes and Connect #155

Closed
lkysow opened this issue Oct 16, 2019 · 6 comments
Closed

Health checks, Kubernetes probes and Connect #155

lkysow opened this issue Oct 16, 2019 · 6 comments
Labels
area/connect Related to Connect service mesh, e.g. injection theme/health-checks About Consul health checking type/enhancement New feature or request

Comments

@lkysow
Copy link
Member

lkysow commented Oct 16, 2019

Currently the Consul Connect integration doesn't hook into Kubernetes health/liveliness probes nor do we provide a way to register Consul health checks. Thus when a Pod goes unhealthy we still route to it.

@lkysow lkysow added area/connect Related to Connect service mesh, e.g. injection type/enhancement New feature or request theme/health-checks About Consul health checking labels Oct 16, 2019
@pbusko
Copy link

pbusko commented May 19, 2020

We're experiencing the same issue here, in our use case TCP health check is not enough. Without integration with k8s readiness/liveness probes requests are frequently being routed to dead pods.

@lkysow
Copy link
Member Author

lkysow commented May 19, 2020

Thanks Pavel, what do you think about us automatically hooking into the kube readiness probes?

@pbusko
Copy link

pbusko commented May 20, 2020

Thanks for your response @lkysow .

Readiness probe hook would be the best option imo. Liveness probe by it's nature doesn't guarantee that pod is ready to accept traffic (at least not always) while readiness probe hook could help to leverage native k8s capabilities without need to duplicate health checks (k8s + consul). Also in such case when not all services deployed in kubernetes are connect-enabled this approach allows to keep consistent way of health check definitions.

@jaredcurtis
Copy link

Hello,

We're investing using Consul connect for our service mesh and have seen this issue as well. We're heavy users of health checks for our existing consul service discovery. Does any workaround exist enable consul to check the health of a connect enabled service?

@lkysow
Copy link
Member Author

lkysow commented Jun 10, 2020

The only workaround I can think of right now is to add a check yourself via consul's api for each connect pod.

@lkysow
Copy link
Member Author

lkysow commented Dec 1, 2020

Health checks have now been released! We now sync Kubernetes probes to Consul. See https://www.consul.io/docs/k8s/connect/health. This was released in consul-helm v0.26.0.

@lkysow lkysow closed this as completed Dec 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connect Related to Connect service mesh, e.g. injection theme/health-checks About Consul health checking type/enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants