rke2 kube-proxy configmap missing

[bpopovich44]

I am running rke2 on a HA cluster. Three master nodes with three worker nodes. Kubernetes version 1.24.10. Through the Rancher UI I installed Prometheus. All the pods are running fine, but when I look at events, I see a Warning my liveness/readiness probes are failing for the prometheus-node-exporters, but only on the worker nodes. Through further investigation, and I am not sure if this is related, in the ""pushprox-ingress-nginx-client"" pod, I am getting the error:

level=info ts=2023-08-14T23:52:22.097Z caller=main.go:232 msg=“Got scrape request” scrape_id=1dfc31ba-14fc-47f6-b3ae-59bc6b4e5aa4 url=http://192.168.121.7:10254/metrics
level=error ts=2023-08-14T23:52:22.098Z caller=main.go:101 err=“failed to scrape http://127.0.0.1:10254/metrics: Get “http://127.0.0.1:10254/metrics”: dial tcp 127.0.0.1:10254: connect: connection refused”
level=info ts=2023-08-14T23:52:22.099Z caller=main.go:113 msg=“Pushed failed scrape response”

Looking through the Prometheus installation documentation, it says to change the metricsBindAddress to 0.0.0.0:10249 from 127.0.0.1:10249 in the kube-proxy config.

"""KubeProxy
The metrics bind address of kube-proxy is default to 127.0.0.1:10249 that prometheus instances cannot access to. You should expose metrics by changing metricsBindAddress field value to 0.0.0.0:10249 if you want to collect them.

Depending on the cluster, the relevant part config.conf will be in ConfigMap kube-system/kube-proxy or kube-system/kube-proxy-config."""

I do not see a kube-proxy configmap. I have been searching through forums and it appears some people see this configmap and some don't. Is there another place this setting is set or how can I change the metricsBindAddress in RKE2.

Thank you!

[brandond]

RKE2 doesn't use configmaps for component configuration. You can pass args to kube-proxy via RKE2's config.yaml on each node:

kube-proxy-arg:
  - metrics-bind-address=0.0.0.0:10249

[bpopovich44]

Thank you Brandon, you have been alot of help with my setup. I will give this a try.

[Arahcdop]

@brandond how to update the provisioned worker node with those configurations.

[murand78]

I had the same issue, solved reading

rancher/rancher#39390

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: rke2-ingress-nginx
namespace: kube-system
spec:
valuesContent: |-
controller:
metrics:
enabled: true
containerPort:
http: 80
https: 443
metric: 10254
hostPort:
ports:
http: 80
https: 443
metric: 10254