TLS is disabled for node_exporter

[jsturtevant]

What happened?
I was looking through the specs generated and the found the message:

kubectl logs node-exporter-cbgwz  
evel=info ts=2021-11-04T22:21:56.153Z caller=tls_config.go:191 msg="TLS is disabled." http2=false

even though secure-listen-address is set

kube-prometheus/manifests/node-exporter-daemonset.yaml

Line 57 in 31d86b7

- --secure-listen-address=[$(IP)]:9100

from what I can tell you have to use --web.config.file to get tls: https://github.com/prometheus/exporter-toolkit/blob/master/docs/web-configuration.md and prometheus/node_exporter#1277 and I wasn't able to find the secure-listen-address in the node-exporter code.

Did you expect to see some different?

How to reproduce it (as minimally and precisely as possible):
deploy the manifests from main branch and look at logs of node_exporter pod

Environment
k8s

• Prometheus Operator version:

  Insert image tag or Git SHA here

• Kubernetes version information:

  kubectl version

• Kubernetes cluster kind:

  insert how you created your cluster: kops, bootkube, tectonic-installer, etc.

• Manifests:

insert manifests relevant to the issue

• Prometheus Operator Logs:

Insert Prometheus Operator logs relevant to the issue here

• Prometheus Logs:

Insert Prometheus logs relevant to the issue here

Anything else we need to know?:
related to #469 and #179

[paulfantom]

kube-prometheus is deploying node-exporter with kube-rbac-proxy sidecar. Sidecar is responsible for RBAC and TLS termination.

We are using this architecture for historical reasons as node_exporter did have TLS termination capabilities before. It might be good to consider updating to use TLS termination in node_exporter itself.

[jsturtevant]

Thanks for the clarification. If either of the other issues linked covers this feel free to close.

[raptorsun]

If we move TLS termination to node-exporter, can kube-rbac-proxy passthrough the TLS connection to upstream node-exporter?
I'm thinking to add an option to move TLS termination to node-exporter while kube-rbac-proxy takes care of access control only.

[paulfantom]

@raptorsun kube-rbac-proxy realizes 2 functions: TLS termination and access restriction. As such moving TLS termination to node_exporter won't allow us to remove kube-rbac-proxy sidecar.

Additionally enabling TLS termination in node_exporter means something would need to provision those certificates. Maybe cert-manager as in #179?

[raptorsun]

What is the feature required from this issue? Is it an option to expose HTTPS port of node-exporter and disable kube-rbac-proxy?
Excuse me plz but I am a little bit confused 😅

[paulfantom]

The issue is about removing kube-rbac-proxy, using node_exporter buillt-in mechanisms for TLS termination and auth. It needs a bit of exploratory work to determine if this is even possible.

[raptorsun]

ok, I will check its feasability first and see if we can proceed to add this feature.

[raptorsun]

Feasibility check result:
Node Exporter is able to terminate TLS and authenticate client by its certificate. It can let pass the client certificates issued by a certain CA while blocking the clients with certificates issued by other CAs. However, it is not capable to determine the role of a client as kube-rbac-proxy does.

The diagram below shows the maximum we can do without kube-rbac-proxy.

This diagram shows how kube-rbac-proxy is used to verify the role of a client with a certificate.

[philipgough]

Just to point out, TLS in node_exporter is marked as experimental as per https://github.com/prometheus/node_exporter#tls-endpoint