strimzi-drain-cleaner during Azure cluster upgrade

[almightykid]

Hello there,

We have recently adopted drain cleaner in our platform but we are seeing strange behavior when it comes to upgrading the AKS cluster. Our setup consists of 3 different node pools each in different AZs and with one node, to guarantee that we have full disponibility in any case. Drain cleaner is running 1.1.0 (legacy mode), operator 0.40 and kafka 3.7.0.

The point is that once the AKS cluster upgrade starts the three nodes are drained at the same time, kafka and zookeepers pods get evicted sequentially to the new nodes but the problem is that once they are running in the upgraded nodes, new nodes are provisioned again and the eviction process starts once again. For the first iteration, we see some warnings in Azure console regarding the nodes not being drained due to PDBs but why does this happen? shouldn't it work just in the first iteration?

I am not sure if this should be asked here but thanks in advance.

[scholzj]

That sounds more like a question on Azure about what is going on in there. The Drain Cleaner and the Strimzi Cluster Operator just react to the eviction requests we get through the Kubernetes APIs.

[almightykid]

Okay, thanks anyway :)