ALB Report 504 errors when every autoscaler decrease node size

[ibalat]

Which component are you using?:

• autoscaler: 1.29.0
• alb controller: 2.7.1

What k8s version are you using (kubectl version)?:
v1.29.0

What environment is this in?:
AWS EKS

What did you expect to happen and instead happened?:
I use prestop hook to prevent 50x errors but when autoscaler run to decrease nodes, ALB report 504 and Target connection errors.

For example, autoscaler start to decrease node size at ~23:15 and ~02:10 and alb start to report 503, 504 and target connection errors at same time. Every night and light hours, I see same problem.

[IMPORTANT] I manually restarted, scaled and deployed all apps to see same errors but no any problem when do any operation. Because I use prestop. Errors coming only when autoscaler decrease node size, not increase them.

How to reproduce it (as minimally and precisely as possible):

• Pods have preStop 35s
• Graceful shutdown time (terminationGracePeriodSeconds) is 60s
• ALB deregistration delay is 20s

[ibalat]

hi @sftim , please can you help for this issue

[adrianmoisey]

/area cluster-autoscaler