v2.9: k8s version upgrade not completing successfully

[cpinjani]

Rancher version:

Rancher version: v2.9-d2fa8a7d6116c48a814ee7983af522d614461e49-head
eks-operator: v1.9.0-rc.9

Cluster Type: Downstream EKS cluster

Describe the bug:
While performing version upgrade for cluster it reverts back to old version the update does not completes successfully, probably due to race condition.

Example screenshot:

Workaround:
Edit cluster and select the upgraded version after the error is seen, save the config.

Logs:

time="2024-07-06T12:53:32Z" level=info msg="waiting for cluster [c-2cfc2] to finish creating"
time="2024-07-06T12:54:02Z" level=info msg="waiting for cluster [c-2cfc2] to finish creating"
time="2024-07-06T12:54:33Z" level=info msg="cluster [c-2cfc2] created successfully"
time="2024-07-06T12:55:02Z" level=info msg="waiting for cluster [c-2cfc2] to update nodegroups [group1]"
time="2024-07-06T12:55:02Z" level=info msg="waiting for cluster [c-2cfc2] to update nodegroups [group1]"
time="2024-07-06T12:55:32Z" level=info msg="waiting for cluster [c-2cfc2] to update nodegroups [group1]"
time="2024-07-06T12:56:03Z" level=info msg="waiting for cluster [c-2cfc2] to update nodegroups [group1]"
time="2024-07-06T12:56:33Z" level=info msg="waiting for cluster [c-2cfc2] to update nodegroups [group1]"
time="2024-07-06T12:57:04Z" level=info msg="cluster [c-2cfc2] finished updating"
time="2024-07-06T13:06:13Z" level=info msg="updating kubernetes version for cluster [c-2cfc2]"
time="2024-07-06T13:06:16Z" level=info msg="updating kubernetes version for cluster [c-2cfc2]"
time="2024-07-06T13:06:17Z" level=info msg="cluster [c-2cfc2] finished updating"
time="2024-07-06T13:06:18Z" level=info msg="updating kubernetes version for cluster [c-2cfc2]"
time="2024-07-06T13:08:04Z" level=info msg="waiting for cluster [c-2cfc2] to finish updating"
time="2024-07-06T13:08:04Z" level=info msg="waiting for cluster [c-2cfc2] to finish updating"
time="2024-07-06T13:08:34Z" level=info msg="waiting for cluster [c-2cfc2] to finish updating"
time="2024-07-06T13:09:05Z" level=info msg="waiting for cluster [c-2cfc2] to finish updating"
time="2024-07-06T13:09:35Z" level=info msg="waiting for cluster [c-2cfc2] to finish updating"
time="2024-07-06T13:10:05Z" level=info msg="waiting for cluster [c-2cfc2] to finish updating"
time="2024-07-06T13:10:35Z" level=info msg="waiting for cluster [c-2cfc2] to finish updating"
time="2024-07-06T13:11:06Z" level=info msg="waiting for cluster [c-2cfc2] to finish updating"
time="2024-07-06T13:11:36Z" level=info msg="waiting for cluster [c-2cfc2] to finish updating"
time="2024-07-06T13:12:06Z" level=info msg="waiting for cluster [c-2cfc2] to finish updating"
time="2024-07-06T13:12:36Z" level=info msg="waiting for cluster [c-2cfc2] to finish updating"
time="2024-07-06T13:13:07Z" level=info msg="waiting for cluster [c-2cfc2] to finish updating"
time="2024-07-06T13:13:37Z" level=info msg="waiting for cluster [c-2cfc2] to finish updating"
time="2024-07-06T13:14:07Z" level=info msg="waiting for cluster [c-2cfc2] to finish updating"
time="2024-07-06T13:14:38Z" level=info msg="updating kubernetes version for cluster [c-2cfc2]"
time="2024-07-06T13:14:38Z" level=error msg="error syncing 'cattle-global-data/c-2cfc2': handler eks-controller: error updating cluster version: error updating cluster [c-2cfc2] kubernetes version: operation error EKS: UpdateClusterVersion, https response error StatusCode: 400, RequestID: 7e105091-f0a9-4d22-8f5c-e0b79eb75ab7, InvalidParameterException: Unsupported Kubernetes minor version update from 1.29 to 1.28, requeuing"
time="2024-07-06T13:14:39Z" level=info msg="updating kubernetes version for cluster [c-2cfc2]"
time="2024-07-06T13:14:39Z" level=error msg="error syncing 'cattle-global-data/c-2cfc2': handler eks-controller: error updating cluster version: error updating cluster [c-2cfc2] kubernetes version: operation error EKS: UpdateClusterVersion, https response error StatusCode: 400, RequestID: 3d9e57c6-0662-4243-abb8-30d82649eb33, InvalidParameterException: Unsupported Kubernetes minor version update from 1.29 to 1.28, requeuing"
time="2024-07-06T13:14:40Z" level=info msg="updating kubernetes version for cluster [c-2cfc2]"
time="2024-07-06T13:14:40Z" level=error msg="error syncing 'cattle-global-data/c-2cfc2': handler eks-controller: error updating cluster version: error updating cluster [c-2cfc2] kubernetes version: operation error EKS: UpdateClusterVersion, https response error StatusCode: 400, RequestID: ab18463c-644b-4b33-9f39-259db33756e3, InvalidParameterException: Unsupported Kubernetes minor version update from 1.29 to 1.28, requeuing"

PR:

• Prevent from downgrading Kubernetes versions #674

[cpinjani]

Validation passed on below build:

v2.9-81337b95660cba868629d6fe769bd7e8242b5aee-head
eks-operator:v1.9.0-rc.11

Logs:

time="2024-07-17T14:48:35Z" level=info msg="updating kubernetes version for cluster [c-2g5w7]"
time="2024-07-17T14:48:37Z" level=info msg="updating kubernetes version for cluster [c-2g5w7]"
time="2024-07-17T14:48:38Z" level=info msg="cluster [c-2g5w7] finished updating"
time="2024-07-17T14:48:39Z" level=info msg="updating kubernetes version for cluster [c-2g5w7]"
time="2024-07-17T14:49:53Z" level=info msg="waiting for cluster [c-2g5w7] to finish updating"
time="2024-07-17T14:49:53Z" level=info msg="waiting for cluster [c-2g5w7] to finish updating"
time="2024-07-17T14:50:23Z" level=info msg="waiting for cluster [c-2g5w7] to finish updating"
time="2024-07-17T14:50:54Z" level=info msg="waiting for cluster [c-2g5w7] to finish updating"
time="2024-07-17T14:51:24Z" level=info msg="waiting for cluster [c-2g5w7] to finish updating"
time="2024-07-17T14:51:54Z" level=info msg="waiting for cluster [c-2g5w7] to finish updating"
time="2024-07-17T14:52:24Z" level=info msg="waiting for cluster [c-2g5w7] to finish updating"
time="2024-07-17T14:52:55Z" level=info msg="waiting for cluster [c-2g5w7] to finish updating"
time="2024-07-17T14:53:25Z" level=info msg="waiting for cluster [c-2g5w7] to finish updating"
time="2024-07-17T14:53:55Z" level=info msg="waiting for cluster [c-2g5w7] to finish updating"
time="2024-07-17T14:54:25Z" level=info msg="waiting for cluster [c-2g5w7] to finish updating"
time="2024-07-17T14:54:56Z" level=info msg="waiting for cluster [c-2g5w7] to finish updating"
time="2024-07-17T14:55:26Z" level=info msg="waiting for cluster [c-2g5w7] to finish updating"
time="2024-07-17T14:55:56Z" level=info msg="waiting for cluster [c-2g5w7] to finish updating"
time="2024-07-17T14:56:27Z" level=info msg="cluster [c-2g5w7] finished updating"