Are KongIngress health checks needed if using Kubernetes Deployment Probes

[MartyBJones]

I am wondering if there is an actual need to setup KongIngress Active/Passive Health checks within a Kubernetes cluster if you have a deployment defined with startup/readiness/liveness probes?
Seems like the deployment probes will handle shifting traffic to pods that are unhealthy without the need of setting up KongIngress active/passive health checks but I want to make sure I am not missing a scenario where they would be needed.

[shaneutt]

Are you running into some issues where health checks are being slow?

[MartyBJones]

We removed the KongIngress Active/Passive checks because we ran into a chicken/egg scenario. We had an Kong Ingress active check that invoked /status/liveness on an API and also had the K8s deployment for this API setup with a liveness probe that also invoked /status/liveness on the the API. If the API ever went down then it could never recover because the pod could never come back online because the /status/liveness call couldn't be made because the pods were taken offline.

We are now just using the liveness probe from the deployment with no KongIngress Active/Passive checks and things work as expected.

I just wanted to make sure I am not missing any potential gotchas for not using KongIngress Active/Passive checks with this configuration.

[shaneutt]

This sounds like something that's more generally problematic and that we should see if we can make improvements for 🤔

Would you be open to creating an issue for this and providing more in-depth details, perhaps even a reproduction environment using something like kind or minikube if you have the time?

Alternatively if you could post more of the details right here, I can convert this discussion into an issue, if you prefer.

[MartyBJones]

I certainly can try to setup a reproduction environment in minikube once I get time.
So do you think there is a need to have "liveness" checks at the deployment level in K8s and within the KongIngress? They seem to be overlapping so that is why we removed the KongIngress checks.

[shaneutt]

It's really hard to answer this generally, because the KongIngress checks are a bit esoteric: they're always very environment specific. The best general advice I can give is that the KongIngress checks are effectively there for the case that Kubernetes checks are not viable, and given what I understand about your situation so far it may be OK to drop them. But to answer the question more fully I would need to understand better exactly why these checks were implemented for in the first place.