Lens Metrics get wrong values of CPU, RAM

[karkill]

Describe the bug
I use the m6i.large instance (AWS), it is in the general purpose family with 2 vCPUs, 8.0 GiB RAM but in Lens I see it double (4 vCPUs, 15.265 GiB RAM)
Screenshots

Environment (please complete the following information):

• Lens Version: 2023.3.21022-latest
• OS: MACOS

[Nokel81]

Do you have multiple prometheus installations installed on this cluster?

[karkill]

I have only one prometheus installed by kube-prometheus-stack

[massimeddu-sj]

Hi have a similar problem with POD memory metric.

At pod level I see the the double of the real value, while at container level I see the right value. (The pod has just one container). See attached screenshot. I also have a Prometheus stack installed with kube-prometheus-stack.

Lens version: 2023.3.71735-latest
OS: Linux

It looks a regression with the latest version of Lens, because with older version of Lens I see the correct value.

[kyleli666]

@jweak my Prometheus is bitnami/kube-prometheus chartVesion8.3.12 appVersion0.63.0
I have POD memory metric doubled, while container memory is correct. Node metrics are also correct.
My cluster is EKS v1.24
this bug happen with all deployments
the metrics were correct in Lens Extension API v6.2.5

[andrezaycev]

+1

[saad-littera]

+1

[centromere]

These are the values being summed together:

Related code:

lens/packages/core/src/main/prometheus/operator-provider.injectable.ts.ts

Lines 79 to 80 in 4b1d740

	case "memoryUsage":
	return `sum(container_memory_working_set_bytes{pod=~"${opts.pods}", namespace="${opts.namespace}"}) by (${opts.selector})`;

The problem may have been introduced in this commit.

[Nokel81]

@centromere Thanks for this. I will add a way to specify which version to use.

[jseletz-rave]

+1 - I am seeing double the CPU and Memory requests for containers than what are actually requested.

[7wingfly]

I've noticed that it starts off with the correct metrics when the nodes are created but then suddenly doubles everything after a few minutes

[vilianvenkov]

+1

[jweak]

Hey guys, I'm unable to reproduce this issue with https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack installed. Could you guys share a bit more about your setup?

What kube version are you using? Which version of the kube-prometheus-stack are you using? Does this bug happen with all deployments? Are you all having both node and pod metrics doubled?

No need to post anything sensitive but some example would go a long way fixing, thanks!

[jseletz-rave]

I am running with the "Builtin Metrics Provider" automatically installed by Lens via the cluster settings. K8s 1.23. All deployments. Both Node and pod metrics are doubled.

[lghinet]

hello,
I have kube-prometheus-stack-45.26.0 installed on a new AKS 1.26.3 (running
mcr.microsoft.com/oss/kubernetes/metrics-server:v0.6.3 in 2 replicas )

pod memory is doubled

[karkill]

Hello bro,
I use kube-prometheus-stack version 45.4.0 (app version v0.63.0) in EKS 1.23.
Errors: CPU, RAM of nodes is doubled.
Lens: 2023.5.120950-latest
I have metrics-server run with 2 replicas but I try scale to 1 but still errors.

[AleksuKey]

Same problem here!

[jweak]

Alright, I think there are two different issues. One with double pod metrics which is from the referred commit and double node metrics which is something different. We will probably have a setting that let's you change the query described in this PR #7777

The node double metrics is still a bit unclear why this happens. The query has not changed in a while.

[JaySeletz]

Any progress on this? It's been broken for quite a while?

[gavinclarkeuk]

Not sure if it is relevant, but I'm seeing an issue where the requests for Succeeded pods (e.g. from Jobs) are being included in the Node level metrics. This means the node level graphs are always overstating the current usage.

[karkill]

Today, when I checked kube_node_status_capacity in prometheus, I saw another data from kubecost. After uninstalling it, everything was back to normal. I'll close this issuses now. Thank for help.

[sawaca96]

Maybe if I use metrics-server and kube-state-metrics together, it will double?

[tonyemiliano]

I’m using Lens 2023.9.290703-Latest, Kube-Prometheus-stack version: 51.2.0
appVersion: v0.68.0
Im using GKE 1.24.15-gke.1700

I installed in the "monitoring" namespace Kube-Prometheus-stack, I’m having problems only in the visualization of the graphics of the nodes. Doesn’t seem to double the value, doesn’t make much sense to me.

All nodes there have the wrong memory value relative to the graph when we click on the node that is where it seems to be right.

I’ve removed everything from namespaces that had other Prometheus installations. Everything seems to work perfect. Minus the memory graphics on the nodes page.

Strange that in my other staging cluster with the same settings is working.

[catemiko]

I’m using Lens 2023.9.290703-Latest, Kube-Prometheus-stack version: 51.2.0 appVersion: v0.68.0 Im using GKE 1.24.15-gke.1700

I installed in the "monitoring" namespace Kube-Prometheus-stack, I’m having problems only in the visualization of the graphics of the nodes. Doesn’t seem to double the value, doesn’t make much sense to me.

All nodes there have the wrong memory value relative to the graph when we click on the node that is where it seems to be right.

I’ve removed everything from namespaces that had other Prometheus installations. Everything seems to work perfect. Minus the memory graphics on the nodes page.

Strange that in my other staging cluster with the same settings is working.

That issue may occur when you have more than one kube-prometheus kubelet in namespace kube-system. You can see all services in kube-system by running kubectl -n kube-system get service -o wide

[hyp3ri0n-ng]

Anyone know why this issue is marked as closed? I'm still seeing the same on my lens installation version 2024.3.70925-latest) with a desktop pro subscription. In fact it seems when I close and open lens everything has doubled again.... one of my pods is now using 90GB of memory (it's actually using 6-9)...

Would be good to know if there is anything that's being worked on with this issue. If not I'm going to have to cancel that subscription :-/ and I really don't want to. Lens has all the info I need and it's presented REALLY nicely. The big problem is that it just uhh, doesn't work....