How to update proxy settings for running cluster?

[zulhfreelancer]

I have a Talos v1.5.3 cluster consists of 1 control plane and 1 worker. I'm trying to set --metrics-bind-address=0.0.0.0:10249 flag to the kube-proxy pods (docs).

According to ProxyConfig section, this can be done using extraArgs field.

So, here is my controlplane.yaml file:

# lines removed for brevity
cluster:
  proxy:
    image: registry.k8s.io/kube-proxy:v1.28.2 
    extraArgs:
      metrics-bind-address: "0.0.0.0:10249"

When I do talosctl apply-config --file controlplane.yaml on existing cluster, the kube-proxy container args didn't get updated.

However, when I do talosctl apply-config --file controlplane.yaml on new cluster, it worked fine.

What else that I've tried but didn't work (in random order):

• use talosctl apply-config with --mode=reboot
• talosctl reboot to reboot the node
• talosctl service kubelet restart
• restart the kube-proxy-* pods in kube-system namespace
• create a patch.yaml file and ran talosctl patch mc -n IP -e IP --talosconfig talosconfig --patch @patch.yaml

Contents of patch.yaml file:

cluster:
  proxy:
    image: registry.k8s.io/kube-proxy:v1.28.2 
    extraArgs:
      metrics-bind-address: "0.0.0.0:10249"

Did I miss something?

[smira]

There's a caveat here - kube-proxy is a bootstrap manifest, it lives fully in Kubernetes. Talos by default doesn't update them, so you need to run talosctl upgrade-k8s to the same Kubernetes version to get things updated.

[zulhfreelancer]

Thanks for the tips. I tried upgrade-k8s but the kube-proxy container args still remain the same. Any idea?

Do I need to run anything before running talosctl upgrade-k8s?

# talosctl upgrade-k8s -n $TALOS_ENDPOINT -e $TALOS_ENDPOINT --to 1.27.4
automatically detected the lowest Kubernetes version 1.27.4
discovered controlplane nodes ["10.0.0.10"]
discovered worker nodes ["10.0.0.11"]
updating "kube-apiserver" to version "1.27.4"
 > "10.0.0.10": starting update
 > "10.0.0.10": machine configuration patched
 > "10.0.0.10": waiting for kube-apiserver pod update
 < "10.0.0.10": successfully updated
updating "kube-controller-manager" to version "1.27.4"
 > "10.0.0.10": starting update
 > "10.0.0.10": machine configuration patched
 > "10.0.0.10": waiting for kube-controller-manager pod update
 < "10.0.0.10": successfully updated
updating "kube-scheduler" to version "1.27.4"
 > "10.0.0.10": starting update
 > "10.0.0.10": machine configuration patched
 > "10.0.0.10": waiting for kube-scheduler pod update
 < "10.0.0.10": successfully updated
updating kube-proxy to version "1.27.4"
 > "10.0.0.10": starting update
updating kubelet to version "1.27.4"
 > "10.0.0.10": starting update
 > "10.0.0.10": machine configuration patched
 > "10.0.0.10": waiting for node update
 < "10.0.0.10": successfully updated
 > "10.0.0.11": starting update
 > "10.0.0.11": machine configuration patched
 > "10.0.0.11": waiting for node update
 < "10.0.0.11": successfully updated
updating manifests
 > processing manifest v1.Secret/kube-system/bootstrap-token-601x2d
 < no changes
 > processing manifest rbac.authorization.k8s.io/v1.ClusterRoleBinding/system-bootstrap-approve-node-client-csr
 < no changes
 > processing manifest rbac.authorization.k8s.io/v1.ClusterRoleBinding/system-bootstrap-node-bootstrapper
 < no changes
 > processing manifest rbac.authorization.k8s.io/v1.ClusterRoleBinding/system-bootstrap-node-renewal
 < no changes
 > processing manifest rbac.authorization.k8s.io/v1.ClusterRole/flannel
 < no changes
 > processing manifest rbac.authorization.k8s.io/v1.ClusterRoleBinding/flannel
 < no changes
 > processing manifest v1.ServiceAccount/kube-system/flannel
 < no changes
 > processing manifest v1.ConfigMap/kube-system/kube-flannel-cfg
 < no changes
 > processing manifest apps/v1.DaemonSet/kube-system/kube-flannel
 < no changes
 > processing manifest apps/v1.DaemonSet/kube-system/kube-proxy
 < no changes
 > processing manifest v1.ServiceAccount/kube-system/kube-proxy
 < no changes
 > processing manifest rbac.authorization.k8s.io/v1.ClusterRoleBinding/kube-proxy
 < no changes
 > processing manifest v1.ServiceAccount/kube-system/coredns
 < no changes
 > processing manifest rbac.authorization.k8s.io/v1.ClusterRoleBinding/system:coredns
 < no changes
 > processing manifest rbac.authorization.k8s.io/v1.ClusterRole/system:coredns
 < no changes
 > processing manifest v1.ConfigMap/kube-system/coredns
 < no changes
 > processing manifest apps/v1.Deployment/kube-system/coredns
 < no changes
 > processing manifest v1.Service/kube-system/kube-dns
 < no changes
 > processing manifest v1.ConfigMap/kube-system/kubeconfig-in-cluster
 < no changes
waiting for all manifests to be applied

[smira]

Either it's already applied to Kubernetes, or Talos machine config hasn't been updated.

Either way, you can check things yourself:

kubectl get deployment kube-proxy -n kube-system -o yaml

talosctl get manifests
...
talosctl get manifests <id> -o yaml

[zulhfreelancer]

Thanks, I tried again and it worked.

# Step 1
talosctl apply-config -e $TALOS_ENDPOINT -n $TALOS_ENDPOINT --file controlplane.yaml

# Step 2
talosctl upgrade-k8s -n $TALOS_ENDPOINT -e $TALOS_ENDPOINT --to 1.27.4

Logs:

# lines removed for brevity
 > processing manifest apps/v1.DaemonSet/kube-system/kube-proxy
--- a/apps/v1.DaemonSet/kube-system/kube-proxy
+++ b/apps/v1.DaemonSet/kube-system/kube-proxy
@@ -2,7 +2,7 @@
 kind: DaemonSet
 metadata:
   annotations:
-    deprecated.daemonset.template.generation: "1"
+    deprecated.daemonset.template.generation: "2"
   labels:
     k8s-app: kube-proxy
     tier: node
@@ -28,6 +28,7 @@
         - --conntrack-max-per-core=0
         - --hostname-override=$(NODE_NAME)
         - --kubeconfig=/etc/kubernetes/kubeconfig
+        - --metrics-bind-address=0.0.0.0:10249
         - --proxy-mode=iptables
         env:
         - name: NODE_NAME

 < applied successfully

The Prometheus Job is happy now: