[SURE-5515] Fleet is not upgrading properly on windows nodes in a cluster with windows workers

[slickwarren]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

After upgrading rancher with an existing cluster that is running rke2 with a windows node from 2.6.8 -> 2.6.9-rc1, (fleet v0.3.11 -> v0.4.0-rc3) , fleet fails to upgrade as it is consistently trying to run on the windows node.

Expected Behavior

Expecting fleet to work in a windows cluster after upgrade

Steps To Reproduce

1. Deploy a windows cluster on rancher v2.6.8, k8s v1.23.10+rke2r1
2. upgrade rancher
3. wait for fleet in the downstream windows cluster to finish upgrading

Environment

- Architecture:
- Fleet Version: v0.3.11
- Cluster: rke2
  - Provider: vsphere
  - Options: with windows worker pool
  - Kubernetes Version: v1.23.10+rke2r1

Logs

MountVolume.SetUp failed for volume "kube-api-access-9w88q" : chown c:\var\lib\kubelet\pods\bfdcae51-9200-440e-89c9-e55b9fb69ef7\volumes\kubernetes.io~projected\kube-api-access-9w88q\..2022_09_20_16_52_32.1316767762\token: not supported by windows

Anything else?

No response

[slickwarren]

still reproducible on v2.7.2-rc7

[aiyengar2]

See rancher/rancher#39372 (comment) for more details

[manno]

I can confirm this is the case by simply looking at the fleet-agent Deployment on any provisioned cluster (here seen from Rancher 2.7.0), which shows that the securityContext.runAsUser is provided while nodeSelector is not set and affinity does not have anything Windows-related

So this deployment doesn't schedule on Windows:

% helm template -n cattle-fleet-system --set apiServerURL=abc fleet-agent charts/fleet-agent
      nodeSelector:
        kubernetes.io/os: linux
      tolerations:
        - key: "cattle.io/os"
          value: "linux"
          effect: "NoSchedule"
          operator: "Equal"

But https://github.com/rancher/fleet/blob/master/pkg/agent/manifest.go#L178-L209 does.
Apparently the toleration, introduced in 2db5cb44, is ineffective?

How do we fix this, do we just add the nodeSelecor or do we add to the built-in affinity rules?

     requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/os
            operator: In
            values:
            - linux 

Adding to affinity could also be done as a workaround via the cluster config.

[manno]

Additionnal QA

Problem

Using fleet in an environment with mixed nodes (linux+windows), would show failures for the windows nodes. Fleet agent cannot run on windows.

Solution

Adding a nodeSelector to select only nodes which have kubernetes.io/os: linux.

Testing

Engineering Testing

Manual Testing

I don't have access to a cluster with windows nodes.

Automated Testing

QA Testing Considerations

Fleet should still start on the linux nodes.

Regressions Considerations

[sbulage]

This is my observation:
I saw in the cluster of 2 Linux node and 1 windows node.
Label on Windows node: beta.kubernetes.io/: windows.
Label on Linux node: kuberenetes.io/os: Linux

I see no fleet-agent pod scheduled on Windows node.