Does EKS BottleRocket support graceful node shutdown?

[carlosjgp]

I've been working on a way to roll out our EKS nodes gracefully and my first attempt was trying to use K8s native support for node graceful shutdown and avoid using any extra infrastructure or deployments

I can see that support for its properties here

• Support for Graceful Node Shutdown in the Kubelet Configuration #1792
• Additional kubelet configuration settings #2930

I tried setting these to shutdown_grace_period=3m and shutdown_grace_period_critical_pod=2m and swapping the nodes between Bottlerocket version 1.14.1 and 1.14.2 while running a simple Nginx deployment (helm create test) with 8 replicas and ingress setup to accept traffic

Then run vegeta to hit this Nginx deployment to ensure the pods were gracefully moved across and respecting the PodDisruptionBudget but there are a lot of failed requests

The nodes are supposed to be tainted with node.kubernetes.io/not-ready:NoSchedule but I didn't see this happening either

I'm using:

• EKS K8s 1.27
• BottleRocket AMI from SSM parameter /aws/service/bottlerocket/aws-k8s-1.27/x86_64/${var.bottlerocket_version}/image_id
• AWS Autoscaling group Instace refresh to roll out the nodes when the AMI id changes

Has someone seen this before?

I could provide more details from Cloudwatch, Container or OS logs and K8s events

[stmcginnis]

Hey @carlosjgp.

Yes, graceful shutdown should work with Bottlerocket. I'm wondering about your configuration here and what could be causing the failed requests.

Nginx is not something that would take awhile to shut down, so I'm not sure graceful shutdown settings would even be coming into play here. Stopping an Nginx pod usually happens quite fast, so I wouldn't expect to see the grace period being used for termination.

Can you describe your deployment a little more? From what I understand, you have one or more nodes running, with the nginx pods running on each node. You then attempt to use an auto scaling group to migrate from the previous release of Bottlerocket to the current release. This adds new nodes that join the cluster. Then you should see new nginx pods being scheduled on to the new nodes, and once running the pods on the old nodes being terminated before remove them from the cluster.

Can you confirm you have a Service created to expose those pods through a static endpoint?

Anything else unusual with your setup (custom scheduler or anything like that)?

[carlosjgp]

Sorry but I don't think that using the AWS LB controller affects the Kubernetes native feature
https://kubernetes.io/docs/concepts/architecture/nodes/#graceful-node-shutdown

The Graceful node shutdown feature depends on systemd since it takes advantage of systemd inhibitor locks to delay the node shutdown with a given duration.

Once systemd detects or notifies node shutdown, the kubelet sets a NotReady condition on the Node, with the reason set to "node is shutting down". The kube-scheduler honors this condition and does not schedule any Pods onto the affected node; other third-party schedulers are expected to follow the same logic. This means that new Pods won't be scheduled onto that node and therefore none will start.

Native K8s graceful shutdown should work with just Autoscaling groups

• https://blog.skouf.com/posts/enabling-graceful-node-shutdown-on-eks-in-kubernetes-1-21/

Maybe you are thinking about the EKS node groups... those have their own, AWS built, node cordon + drain and they require the LB controller since they don't support other ways of ingress

[stmcginnis]

I was actually thinking graceful shutdown isn't coming in to play here. As I mentioned, nginx shutdown is very quick, so I would be surprised if k8s needed to wait for the configured grace time before the pod could be terminated.

It seems more like an issue with the service routing not picking up the new node and pod instances quickly enough for the requests to fail. That's why I was thinking this might have something to do with ELB or the lack of a Service endpoint.

It might be difficult with how quickly things happen, but it would be interesting to see what pods are present, and on which nodes, during that window when you are seeing request failures.

Maybe someone else has some ideas of other things to check here, but it doesn't seem to me like the shutdown_graceperiod* settings are a factor here.

[carlosjgp]

I might be interpreting the K8s graceful node shutdown feature

What I'm expecting is:

• Node receives shutdown signal
• Kubelet SystemD unit sees it and uses SystemD inhibition locks to halt the shutting-down process
• Kubelet sets a NotReady condition on the Node, with the reason set to "node is shutting down".
• All PODs in the node are now "NotReady" because of the node status change (no traffic is forwarded here)
• Kubelet also begins terminating any Pods that are running locally. PDB should be respected
• No PODs are left in the node but the ones belonging to Deamonsets
• Kubelet releases the SystemD lock
• Node is terminated

What I'm seeing:

• Node receives shutdown signal
• Node and PODs are terminated simultaneously
• Terminated PODs are scheduled on a different node

It seems more like an issue with the service routing not picking up the new node and pod instances quickly enough for the requests to fail. That's why I was thinking this might have something to do with ELB or the lack of a Service endpoint.

In my case, the ALB is registered against the ASG this means that instances being terminated are removed and added from the ALB by the ASG based on the instance state. The ASG would add any instance InService to the ASG and remove the Unhealthy ones. When a node changes to NotReady it immediately fails to pass the ALB health check based on IngressNginx /healthz endpoint and that instance is removed from the pool.

As you can see here the IngressNginx PODs do have a pod lifecycle to wait to be terminated and drain their inflight requests
https://github.com/kubernetes/ingress-nginx/blob/afd1311f8529c21fdf6621bf683bec814e698f1d/charts/ingress-nginx/values.yaml#L734-L738

I've ended up using AWS Termination Handler + SQS queue for ASG to manage the graceful shutdown and that works but it's the ASG controlling the graceful shutdown, not the K8s native feature. It would be nice to not have an extra Daemonset only for this when there is a

[stmcginnis]

You're absolutely right @carlosjgp. It looks like Bottlerocket is missing systemd-logind that is needed for this functionality. I've opened #3305 based off of this and will look in to what would need to be done.

[carlosjgp]

Thanks for your help @stmcginnis I was unsure if there was another path for the inhibitors to work and maybe my setup was faulty or my assumption of how the graceful shutdown work was wrong

Loads of ❤️ to the Bottlerocket team for these AMIs