Long running jobs exiting on docker 23+ #17499
Labels
hcc/cst
Admin - internal
stage/accepted
Confirmed, and intend to work on. No timeline committment though.
theme/driver/docker
type/bug
Projects
Milestone
Nomad version
Nomad v1.5.5
BuildDate 2023-05-05T12:50:14Z
Revision 3d63bc6
Operating system and Environment details
Ubuntu 20.04
Issue
Currently a bug seems to exist with docker version 23+ that long running jobs being shutdown are exiting before the respected shutdown times. Docker 20.10 behaves as expected.
Reproduction steps
Create a long running job, and set
kill_timeout = "1h"
Scale your service down to 0 to emulate it gracefully shutting down. It should be terminated in around 10 minutes as it seems that the request to the docker shutdown endpoint is timing out.
Expected Result
Gracefully stops
Actual Result
Job is terminated early
Nomad Client logs (if appropriate)
I've also opened an issue with docker about this, as it seems there was an API change made to the stop endpoint and i'm not sure if it's expected behavior or not.
moby/moby#45731
The text was updated successfully, but these errors were encountered: