BREAKING CHANGE: replace node.kubernetes.io/unschedulable with a Karpenter-specific taint

[njtran]

Fixes #N/A

Description
Design Doc: #585

Karpenter will now use the karpenter.sh/disrupting taint instead of node.kubernetes.io/unschedulable taint when deprovisioning nodes. Karpenter only adds this taint once it has begun spinning up replacement nodes for deprovisioning actions, and subsequently deleting the candidate node.

• Karpenter would add/remove this taint, assuming no other agent had been managing this taint.
• This fixes an issue where in rare cases where Karpenter restarted while executing a deprovisioning action, Karpenter can leave nodes that were meant to be deprovisioned as cordoned.
• This breaks some users with pods that tolerate the node.kubernetes.io/unschedulable taint or tolerate all keys: a * toleration. If users want to ensure their daemonsets and other pods that had this toleration are not evicted as part of termination, they'll need to add this toleration to their pods.
• This also modifies the cluster state logic to remove the stateNode.MarkedForDeletion variable. Karpenter can now rely on the existence of the karpenter.sh/disrupting taint to know when a node has been marked for deletion.

How was this change tested?

• make presubmit
• make apply && manual test

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[njtran]

Adding the blocked label so we can first vet if changing this taint right now is what we want.

[jonathan-innis]

Looks good! Just a few thoughts.

[jonathan-innis]

LGTM 🚀

[jonathan-innis]

LGTM 🚀

[coveralls]

Pull Request Test Coverage Report for Build 6500758663

• 54 of 64 (84.38%) changed or added relevant lines in 4 files are covered.
• 4 unchanged lines in 1 file lost coverage.
• Overall coverage decreased (-0.006%) to 82.189%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/controllers/deprovisioning/controller.go	43	53	81.13%

Files with Coverage Reduction	New Missed Lines	%
pkg/controllers/deprovisioning/controller.go	4	80.95%

Totals
Change from base Build 6500425875:	-0.006%
Covered Lines:	9077
Relevant Lines:	11044

---

💛 - Coveralls