-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature Request] Add option to nullify docker container healthchecks #5310
Comments
Also please consider the other way around, using the status of the |
I agree with @rgl. |
…thcheck This PR adds a docker driver task configuration setting for turning off built-in HEALTHCHECK of a container. References) https://docs.docker.com/engine/reference/builder/#healthcheck https://github.com/docker/engine-api/blob/master/types/container/config.go#L16 Closes #5310 Closes #14068
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Description
When building docker containers the option exists to declare a HEALTHCHECK during the build process. Nomad doesn't use this information at all and declares its own checks to monitor container health.
Essentially this means that unless you are running more than one scheduler to manage your containers there is no reason to have these health checks running.
Thus, it would be nice to have the ability to disable native healthchecks at runtime if the containers being scheduled had a useless but resource consuming HEALTHCHECK defined.
Use case
A long time ago, possibly before we even started using nomad to manage our clusters, I made the extremely naive decision to throw in a
In some of our Dev and Test environments we have up to 70 jobs running per node. Some of which contain code which takes a long time to start up. looking at the processes running on a node if we have just drained one of these giant nodes and have 60+ jobs starting at once reveals literally thousands of
curl
commands building up before things stabilise. Furthermore, after some time, when a couple of jobs on a node have become unhealthy the docker daemon seems to lock up and nomad is unable to remove the now-unresponsive container.We didn't actually notice that this was an issue previously because our nodes were a more sensible size so the cumulative impact went unnoticed.
With this issue our current way forward is likely going to be to just throw another layer onto currently running versions of containers with
as the only change. However one of the fun things about working in a large organisation is that I'm sure that some of the technical owners are going to want to manually regression test the change because it's a change they don't understand to their container.
Examples
The docker CLI provides a --no-healthcheck option at run-time. The API allows for NONE to be passed in to disable any predefined health checks. From memory, Kubernetes disables docker health checks all the time since ~mid-2017ish.
So this seems like it should be incredibly simple to do at the point in time that container configuration is being put together before the container starts.
My assumption would be that it would be better to default to leaving docker health checks as they are, on the off chance that the results are being used by someone for something, but to provide an option in the task config to disable native health checks if desired.
The text was updated successfully, but these errors were encountered: