kubeadm upgrade fails because of virtual nodes

[dimm0]

• Nodes have no version, which confuses the upgrade script

[upgrade/version] FATAL: the --version argument is invalid due to these errors:

	- couldn't parse kubelet version

Can be bypassed if you pass the --force flag
To see the stack trace of this error execute with --v=5 or higher

• Nodes go on and off, which makes install script unhappy ("control plane is not ready")
• In the end upgrade script fails:

[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upgrade/postupgrade] Applying label node-role.kubernetes.io/control-plane='' to Nodes with label node-role.kubernetes.io/master='' (deprecated)
timed out waiting for the condition
To see the stack trace of this error execute with --v=5 or higher

To finish the upgrades I had to delete all virtual nodes and disable admiralty.

[adrienjt]

1. This looks like kubeadm upgrade, is that right? First of all, is there a way to exclude nodes from the upgrade process? It doesn't make sense to upgrade virtual nodes.

2. Nodes have no version

   Virtual nodes have no status.nodeInfo.kubeletVersion because they could be backed by clusters running multiple versions (e.g., being upgraded).

3. Nodes go on and off

   What do you mean?

4. In the end upgrade script fails

   I wonder what condition timed out waiting for the condition the script refers to. Looking at the snippet, this might be because node-role.kubernetes.io/master= is aggregated on virtual nodes, and the node-role.kubernetes.io/control-plane= label is removed by Admiralty as soon as it's applied. Could be fixed with new target option spec.excludedLabelsRegexp: ^node-role\.kubernetes\.io/master=$.

[dimm0]

1. This looks like kubeadm upgrade, is that right? First of all, is there a way to exclude nodes from the upgrade process? It doesn't make sense to upgrade virtual nodes.

Not that I know of...

3. Nodes go on and off

This is from my previous attempts to upgrade. I think it was hung when nodes were not ready.

4. In the end upgrade script fails

   I wonder what condition timed out waiting for the condition the script refers to. Looking at the snippet, this might be because node-role.kubernetes.io/master= is aggregated on virtual nodes, and the node-role.kubernetes.io/control-plane= label is removed by Admiralty as soon as it's applied. Could be fixed with new target option spec.excludedLabelsRegexp: ^node-role\.kubernetes\.io/master=$.

I'd think that's the reason.

Anyway, I think it's worth adding to docs some recommendations on how to upgrade the cluster (and at least test this)

[adrienjt]

I'd love your help with this:

• test kubeadm upgrade after configuring targets with spec.excludedLabelsRegexp: ^node-role\.kubernetes\.io/master=$
• make sure virtual nodes remain ready during kubeadm upgrade
• add documentation page in operator guide about running kubeadm upgrade in source/management cluster
• consider excluding node-role.kubernetes.io/master= and node-role.kubernetes.io/control-plane= from virtual node label aggregation by default
• add e2e test with kubeadm

[dimm0]

I'm about to do another upgrade.
Tried adding the spec.excludedLabelsRegexp param to 3 of my targets, and 1 of them keep respawning with master label. Other 2 are fine. Any tips?

In that one a federated pod is running

[dimm0]

make sure virtual nodes remain ready during kubeadm upgrade

The control plane goes offline during the upgrade, so can't really do that