Kubernetes Upgrade with Cilium

[checkelmann]

Hi all,

I have installed a Talos Cluster with Cilium following the Hosted Manifest Guide (https://www.talos.dev/v1.6/kubernetes-guides/network/deploying-cilium/#method-3-helm-manifests-hosted-install).

After a while I wanted to switch to the helm install option by removing cilium like kubectl delete -f cilium.yaml, installed cilium using helm and removing the cni config from my controlplane.yaml and worker.yaml.

In the meantime, I upgraded from cilium 14.x to 15 (using helm upgrade).

Now I wanted to update my Kubernetes Version from 1.29 to 1.29.1 but recognized when running the k8s dryrun, that talos want's to revert my helm install to the "old" cilium mainifest version.

talosctl --nodes 192.168.0.50 upgrade-k8s --dry-run

....'

--- a/apps/v1.DaemonSet/kube-system/cilium
+++ b/apps/v1.DaemonSet/kube-system/cilium
@@ -2,11 +2,8 @@
 kind: DaemonSet
 metadata:
   annotations:
-    deprecated.daemonset.template.generation: "3"
-    meta.helm.sh/release-name: cilium
-    meta.helm.sh/release-namespace: kube-system
+    deprecated.daemonset.template.generation: "4"
   labels:
-    app.kubernetes.io/managed-by: Helm
     app.kubernetes.io/name: cilium-agent
     app.kubernetes.io/part-of: cilium
     k8s-app: cilium

         - name: K8S_NODE_NAME
@@ -217,7 +186,7 @@
             fieldRef:
               apiVersion: v1
               fieldPath: metadata.namespace
-        image: quay.io/cilium/cilium:v1.15.0@sha256:9cfd6a0a3a964780e73a11159f93cc363e616f7d9783608f62af6cfdf3759619
+        image: quay.io/cilium/cilium:v1.14.5@sha256:d3b287029755b6a47dee01420e2ea469469f1b174a2089c10af7e5e9289ef05b
         imagePullPolicy: IfNotPresent
         name: config
         resources: {}

I wonder where talos is still getting the old values from?
I also have re provisioned my control planes with the "new" controlplane.yaml where the CNI option is set to none.

I would appreciate any help!

[frezbo]

it should be coming from the machineconfig, the inlineManifests

[checkelmann]

Thank you for your reply. As mentioned I removed them from the yaml files for worker and controlpalne already or do you mean somewhere else?

[smira]

worker nodes don't have manifests in their config (or shouldn't have).

if the manifest appears in the output, it is still in the controlplane node machine configuration

[checkelmann]

Thank you, @smira but it isn't and this is what I don't get where it's still coming from ;)

When you say controlplane node machine configuration you refer to the controlplane.yaml file or anything else?

controlplane.yaml

cluster:
    id: REMOVED # Globally unique identifier for this cluster (base64 encoded random 32 bytes).
    secret: REMOVED # Shared secret of cluster (base64 encoded random 32 bytes).
    # Provides control plane specific configuration options.
    controlPlane:
        endpoint: https://REMOVED:6443 # Endpoint is the canonical controlplane endpoint, which can be an IP address or a DNS hostname.
    clusterName: talos-proxmox-cluster # Configures the cluster's name.
    # Provides cluster specific network configuration options.
    network:
        # The CNI used.
        cni:
            name: none # Name of CNI to use.
            # URLs containing manifests to apply for the CNI.
            #urls:
            #    - none
        dnsDomain: cluster.local # The domain used by Kubernetes DNS.
        # The pod subnet CIDR.
        podSubnets:
            - 10.244.0.0/16
        # The service subnet CIDR.

[checkelmann]

Ok I figured it out. I reapplied the controlplane.yaml to all control planes

 talosctl apply-config -n IP--file controlplane.yaml