fix inter-pod anti-affinity issue

[wenlxie]

This is used to fix:
#50813

Release note:

[scheduler] Fix issue new pod with affinity stuck at `creating` because node had been deleted but its pod still exists.

[k8s-ci-robot]

Hi @wenlxie. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[timothysc]

Please add a test case for this condition, I would recommend an integration test b/c it uses fake-nodes which should be easy to simulate.

Also, please add a release-note on this one.

[anfernee]

I am not sure this is the right fix, otherwise masking errors will cause more harm in future. The biggest questions to be answered here are:

• why node still in cache when it's actually gone? from the current code, we assume it's possible removePod comes after RemoveNode. When all pods are deleted from a node, the node will be removed from the cache. But it's not happening for some reason, which causes this bug. we should figure out why. Node is removed from cache before pod.Spec.NodeName is removed.
• the CacheNodeInfo that predicate uses isn't cached at all. NodeList itself is cached.

  kubernetes/plugin/pkg/scheduler/algorithm/predicates/predicates.go

  Lines 111 to 113 in 58fd063

  	type CachedNodeInfo struct {
  	corelisters.NodeLister
  	}

  Is it intentional? Whatever it is, the name is confusing. If this uses cache, this bug will not happen as well. It only happens when cache is out of sync. But it's a good thing that this help exposing the cache problem.

@wenlxie if you still on this issue, we can work together addressing this issue.

[wenlxie]

@anfernee Thanks for the reply.
It is easy to reproduce this issue, it happens in following scenario:

1. The node had been removed from the cluster,
2. There are pod that used to running on there node still exists
3. A new pod with podAntiAffinity will stuck at pending

You can add a finalizer to the running pod, and then delete its node from k8s, the pod will not get deleted.

[anfernee]

@wenlxie just curious how do you remove a node? I tried drain, cordon and just shut the node down, they all seem to work fine.

[wenlxie]

@anfernee Use kubectl delete node command

[anfernee]

I still cannot reproduce, but I believe @wenlxie 's fix will fix the issue.

[timothysc]

The change seems safe but ideally I would like to see a test case for it.

/lgtm
/approve

[k8s-ci-robot]

@fejta-bot: you can't request testing unless you are a kubernetes member.

In response to this:

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[k8s-ci-robot]

@fejta-bot: you can't request testing unless you are a kubernetes member.

In response to this:

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[k8s-ci-robot]

@fejta-bot: you can't request testing unless you are a kubernetes member.

In response to this:

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[k8s-ci-robot]

@fejta-bot: you can't request testing unless you are a kubernetes member.

In response to this:

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[k8s-ci-robot]

@fejta-bot: you can't request testing unless you are a kubernetes member.

In response to this:

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[k8s-ci-robot]

@fejta-bot: you can't request testing unless you are a kubernetes member.

In response to this:

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[k8s-ci-robot]

@fejta-bot: you can't request testing unless you are a kubernetes member.

In response to this:

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[k8s-ci-robot]

@fejta-bot: you can't request testing unless you are a kubernetes member.

In response to this:

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[k8s-ci-robot]

@fejta-bot: you can't request testing unless you are a kubernetes member.

In response to this:

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[k8s-ci-robot]

@fejta-bot: you can't request testing unless you are a kubernetes member.

In response to this:

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[k8s-ci-robot]

@fejta-bot: you can't request testing unless you are a kubernetes member.

In response to this:

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[k8s-ci-robot]

@fejta-bot: you can't request testing unless you are a kubernetes member.

In response to this:

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dims]

/retest

[k8s-github-robot]

/test all [submit-queue is verifying that this PR is safe to merge]

[k8s-github-robot]

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.