Add aws-k8s-1.21, vmware-k8s-1.21 variants

[etungsten]

Issue number:
#1611

Description of changes:

For reference, the addition of aws-k8s-1.20 was in #1437.

Version of kubelet packaged: v1.21.2

Author: Erikson Tung <etung@amazon.com>
Date:   Tue Jun 15 11:31:57 2021 -0700

    Add aws-k8s-1.21 variant
    
    Adds aws-k8s-1.21 variant, relinks symlinks in models

Author: Erikson Tung <etung@amazon.com>
Date:   Tue Jun 15 11:55:19 2021 -0700

    Add vmware-k8s-1.21 variant
    
    Adds vmware-k8s-1.21 variant, relinks symlinks in models

Author: Erikson Tung <etung@amazon.com>
Date:   Tue Jun 15 11:30:21 2021 -0700

    packages: add kubernetes-1.21
    
    * No longer need '0001-always-set-relevant-variables-for-cross-compiling.patch'
    * Third party package 'github.com/miekg/dns' version bump included some
    changes in its license file so the hash changed. The license is still
    'BSD-3-Clause'.

0001-always-set-relevant-variables-for-cross-compiling.patch is no longer needed as a result of the following changes in 1.21:

• kubernetes/kubernetes@fd01467
• kubernetes/kubernetes@f19f22b
  We export the KUBE_*_CC variables directly.

Testing done:

• Pods run OK

• Conformance testing with 1.21 control plane and x86_64 aws-k8s-1.21 nodes

• Conformance testing with 1.21 control plane and aarch64 aws-k8s-1.21 nodes

• Conformance testing with 1.21 control plane and x86_64 vmware-k8s-1.21 nodes

• Confirm the EBS CSI driver still works.

• Confirm the vSphere CSI driver still works - (Block volumes only since file volumes are not yet supported on AWS VMC)

• Check kubelet logs and system logs in general, compared them with the aws-k8s-1.20 variant kubelet logs:

There's a new kubelet warning during start up:

Flag --network-plugin has been deprecated, will be removed along with dockershim.

dockershim for kubelet will be fully deprecated by 1.23 which I believe we'll have to address separately at some point.

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[tjkirch]

LGTM pending testing.

[etungsten]

Push above separates the commits into 3, first one adds the kubernetes-1.21 package, one that adds the aws-k8s-1.21 variant, and one that adds the vmware-k8s-variant.

[etungsten]

Push above adds some clarifying comments about kube_cc in kubernetes-1.21.spec

[etungsten]

Push above rebases onto develop

[etungsten]

Push above addresses comments from @tjkirch

[etungsten]

Updated PR description to include testing for vsphere conformance testing, vsphere csi driver testing against a 1.21 cluster control plane.

[etungsten]

Github actions is consistently failing checks. I added three more jobs as part of this PR. There are now a total of 18 jobs to run per PR. It seems like one or two of the jobs always times out for some reason.

[zmrow]

💸

[etungsten]

v1.21.2 just got released. I'll update the packaged kubelet version soon.

[tjkirch]

LGTM, assuming 1.21.2 was retested.

[etungsten]

Push above updates kubelet to v1.21.2. Pods run fine.

Build checks keep failing due to network timeouts etc. Failures are not related to the changes.

[etungsten]

Rebases onto develop to bring in the commit that fixes CI.