[BUG] [v2.9.1] Can not upgrade downstream cluster to new RKE template revision

[phoenix-bjoern]

SURE-9052

This issue was initially reported in rancher/rancher#46967, but maybe is only a bug in the UI.

Rancher Server Setup

• Rancher version: v2.9.1
• Installation option (Docker install/Helm Chart): Docker install

Information about the Cluster

• Kubernetes version: v1.28.9
• Cluster Type (Local/Downstream): Custom Downstream

User Information

• What is the role of the user logged in? Admin

Describe the bug
Rancher has been upgraded from v2.8.4 to v2.9.1. A new RKE1 template revision has been cloned to prepare a K8S upgrade from v1.28.9 to v1.29.7. When editing an existing downstream cluster the new RKE1 template revision can not be saved.

At first the UI looks normal:

However, when a RKE template is selected...

...form elements and the save button disappear:

It is not possible to save the changes. Hence it is not possible to upgrade a cluster which is based on the RKE1 template.

To Reproduce

1. Create a new revision of a RKE1 template.
2. Edit an existing downstream cluster.
3. Select the new RKE1 template revision.

Result
The form elements for "Cluster Options" disappear, also the "Save" button.

Expected Result
The form updates the information in the "Cluster Options" section according to the RKE1 template settings and the new revision can be saved to initiate the downstream cluster upgrade.

[phoenix-bjoern]

Can still reproduce the issue with Rancher v2.9.2.

[jiaqiluo]

I was able to reproduce this issue on a Rancher v2.9.1 Docker installation.

Steps:

1. Create an RKE cluster template, leaving all values as default except setting the Kubernetes version to a non-latest version, such as 1.28.x.
2. Create a custom RKE cluster using this template. Register a single node with all roles to the cluster and wait for the cluster to become active.
3. Create a new revision of the same cluster template, only changing the Kubernetes version to a higher version like 1.29.
4. Attempt to upgrade the existing custom cluster using the new template revision.

Results:

• As reported, sections like Kubernetes options, private registries, and advanced options disappear from the UI.
• No errors appeared in the backend logs or browser console.

This issue does not occur on Rancher v2.8.8.

I would like to request the UI team to investigate this issue on the UI side.

cc @gaktive

[phoenix-bjoern]

@gaktive Due to this bug it is not possible to update downstream RKE1 clusters. Therefore security of the managed clusters is also affected.
Please reconsider the priority of this issue and the release schedule.
Thanks!

[gaktive]

Could be tied to the recent Ember upgrade. Need to have an engineer repro this.

[ManuelSimon]

Workaround here:

We will be using the Rancher v3 API. It has its own user interface accessible from a web browser. Workaround steps:

• Access the Rancher v3 API of the downstream cluster where you want to upgrade the Kubernetes version: https://[RANCHER-URL]/v3/clusters/[CLUSTER-ID]
  • To easily find the cluster ID, go to Rancher UI -> ☰ -> Cluster Management -> Select the cluster -> On the URL bar, you will see something similar to: https://[RANCHER-URL]/dashboard/c/_/manager/provisioning.cattle.io.cluster/fleet-default/c-xxxxx. The last 7 characters (c-xxxxx) are your cluster ID.
• You will be able to see now the Rancher v3 API. In the top-right corner, you will see an operations window.
  • Click on "Edit".
  • You will see all your cluster's configurations.
  • Help you with Ctrl+F to find the "clusterTemplateRevisionId" value. It should show a select field where you can see all your template revisions. Click on the revision you are interested in to upgrade the Kubernetes version or any other change.
  • Scroll down until you see the "Show Request" button. Click on that button and scroll down again until you see the "Send Request" button. Click on it, and the changes will be applied.

[phoenix-bjoern]

@ManuelSimon Thank for the detailed description! I can confirm that this workaround works and the downstream clusters can actually be updated via API.

[yonasberhe23]

Cluster Type	Test case	Result
RKE1 custom	Verify that the right create/edit pages load	✅
RKE2 custom	Verify that the right create/edit pages load	✅
RKE1 provisioned	Verify that the right create/edit pages load	✅
RKE2 provisioned	Verify that the right create/edit pages load	✅
Generic imported	Verify that the right create/edit pages load	✅
AKS/EKS/GKE imported (1 of the 3)	Verify that the right create/edit pages load	✅
AKS/EKS/GKE provisioned (1 of the 3)	Verify that the right create/edit pages load	✅
RKE1 cluster	verify that when using an RKE1 template users can select new template versions and the form is updated accordingly	✅

Tested in:

• Rancher v2.10-695357ad2fdb58bfce027ecf6ce96e8a4e3ddbab-head
• Dashboard master d5024a1