Cancel a plan from the prepare step

[TallFurryMan]

I would like to know how to cancel/bypass a Plan once it is selected.

My example use case is a system upgrade, say from Buster to Bullseye, then from Bullseye to Bookworm.

• My first Plan is called bullseye-upgrade, thus will apply plan.upgrade.cattle.io/bullseye-upgrade=<version>
• That first Plan matches label plan.upgrade.cattle.io/buster-upgrade with any value.
• So initially I assign label plan.upgrade.cattle.io/buster-upgrade to my nodes in order to trigger the first Plan.
• My second Plan is called bookworm-upgrade, thus will apply plan.upgrade.cattle.io/bookworm-upgrade=<version>
• That second Plan matches label plan.upgrade.cattle.io/bullseye-upgrade with any value.

To make this work, I apply the label plan.upgrade.cattle.io/buster-upgrade or plan.upgrade.cattle.io/bullseye-upgrade with value 0 to my nodes, depending on their distribution, in order to trigger the updates.

You may see the issue approaching: because the version is 0, the Controller decides that the Plan that matches the label is outdated and reapplies it. No problem, I can intercept this in the prepare or upgrade initContainers and not do anything. And the node is labelled with the latest Plan version as a result.

However there seems to be no way to bypass the upgrade initContainer. So even if the Plan does nothing (because its script detects there is nothing to do), the node is cordoned/drained because it might need to be rebooted. That is not a no-go, but could be improved.

So, how do I "bypass" an upgrade and its impact on the cluster node once a Plan is selected?

[dweomer]

To bypass the drain and upgrade containers for a plan you should be able to, from within the prepare init-container, replicate the label on the node that the SUC would apply as if the plan had run to completion successfully.

This could mean that, if in the prepare init-container you can determine whether the upgrade container will do any actual work before it is invoked, e.g. performing the apt-get dist-upgrade in the prepare and then conditionally reboot in upgrade if /host/run/reboot-required exists, then you should be able to side-step both the drain init-container and the upgrade container by setting the plan.upgrade.cattle.io/${SYSTEM_UPGRADE_PLAN_NAME}=${SYSTEM_UPGRADE_PLAN_LATEST_HASH} from within the prepare init-container and then sleeping indefinitely.

The updated label on the node, plan.upgrade.cattle.io/${SYSTEM_UPGRADE_PLAN_NAME}=${SYSTEM_UPGRADE_PLAN_LATEST_HASH}, will cause SUC to think that the plan no longer needs to be applied to the node which will result in any existing jobs (for said plan and node) to be terminated.

[TallFurryMan]

Ah that's a very good idea. At first read I thought the drain or upgrade containers would check the label, but that's not the case in the code. Then I read "sleeping indefinitely" until the pod deadline, and that made sense.

To apply the label in the prepare step, I'd then need a kubectl image, image which is probably able to run the pre-upgrade checks. I will check on the service account accreditations too for minimal privilege.

Thanks!