Deployment scale down has unexpected behavior

[howardjohn]

Steps:

• Run kwok locally, connected to a kind cluster
• Deploy 2 fake kwok nodes with high pods per node
• Deploy fake-pod from docs
• Scale it to 999 and back down

I expect this to be very fast. In practice, it takes a while.

On scale up, I see:

2023-02-10T08:59:01.608120 NAME       READY   UP-TO-DATE   AVAILABLE   AGE
2023-02-10T08:59:01.608332 fake-pod   1/1     1            1           12m
2023-02-10T08:59:02.247084 fake-pod   1/999   1            1           12m
2023-02-10T08:59:02.255063 fake-pod   1/999   1            1           12m
2023-02-10T08:59:02.260787 fake-pod   1/999   1            1           12m
2023-02-10T08:59:50.757661 fake-pod   501/999   501          501         13m
2023-02-10T08:59:50.809376 fake-pod   999/999   999          999         13m

So almost 1 min to scale up.

On scale down, logs are spammed with:

Delete pod                                                                                                                 pod="default/fake-pod-6cf6574478-t6ntb" node="kwok-node-1"
WARN Delete pod                                                     pod="default/fake-pod-6cf6574478-t6ntb" node="kwok-node-1" !BADKEY="pods \"fake-pod-6cf6574478-t6ntb\" not found"
ERROR Failed to finalizers                            pod="default/fake-pod-6cf6574478-sqrh6" node="kwok-node-1" err="the server rejected our request due to an error in our request"

And takes about 1min as well:

2023-02-10T09:00:42.833048 fake-pod   999/1     999          999         14m
2023-02-10T09:00:42.843271 fake-pod   999/1     999          999         14m
2023-02-10T09:01:31.405940 fake-pod   499/1     499          499         15m
2023-02-10T09:01:31.455725 fake-pod   1/1       1            1           15m

The slowness could reasonably be from k8s itself, but the error logs on scale down seem suspicious at least

[wzshiming]

/assign
/kind bug

[wzshiming]

Fixed by #275
/close

[k8s-ci-robot]

@wzshiming: Closing this issue.

In response to this:

Fixed by #275
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.