Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue with labelmaps in prometheus relabel config generating empty strings #823

Closed
eldondev opened this issue Sep 11, 2020 · 1 comment · Fixed by #828
Closed

Issue with labelmaps in prometheus relabel config generating empty strings #823

eldondev opened this issue Sep 11, 2020 · 1 comment · Fixed by #828

Comments

@eldondev
Copy link
Contributor

I hope you don't mind if I err on the side of conciseness in this issue. If it would be better I can edit this to match the github template. The relabel labelmap configs such as are defined here seem to use the nodename as the regex to match. However, the "replacement" component of the "labelmap" instance takes the first regex capture group if I am reading this correctly. Since no capture group is present, the result is an empty label (""="node1.local" for example). This causes alertmanager to complain to me. However in my case the examples that I found had an "instance" label, that I would expect to match the "node name" label. In my case, I believe removing the additional labelmap step was sufficient to generate desired behavior. What was the labelmap intended to do?
before removing relabel:
etcd_debugging_disk_backend_commit_rebalance_duration_seconds_count{="node1.local",instance="node1.local",job="etcd"}
after:
etcd_debugging_disk_backend_commit_rebalance_duration_seconds_count{instance="node1.local",job="etcd"}

If you'd like, I can create a PR just to remove the misbehaving labelmaps.

@dghubble
Copy link
Member

Yeah, that's right. You can remove that labelmap.

In its original form, all node labels were collected. But time series reductions removed that since those labels are seldom valuable (#596) leaving just the name. But the instance is what's used in dashboards.

dghubble added a commit that referenced this issue Sep 13, 2020
* Originally, Kubelet and CAdvisor metrics used a labelmap
relabel to add Kubernetes SD node labels onto timeseries
* With #596 that
relabel was dropped since node labels aren't usually that
valuable. `__meta_kubernetes_node_name` was retained but
the field name is empty
* Favor just using Prometheus server-side `instance` in
queries that require some node identifier for aggregation
or debugging

Fix #823
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants