Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Prometheus etcd target discovery #561

Merged
merged 1 commit into from
Oct 6, 2019
Merged

Conversation

dghubble
Copy link
Member

@dghubble dghubble commented Oct 4, 2019

  • Prometheus was configured to use kubernetes discovery of etcd targets based on nodes matching the node label node-role.kubernetes.io/controller=true
  • Kubernetes v1.16 stopped permitting node role labels node-role.kubernetes.io/* so Typhoon renamed these labels (no longer any association with roles) to node.kubermetes.io/controller=true
  • As a result, Prometheus didn't discover etcd targets, etcd metrics were missing, etcd alerts were ineffective, and the etcd Grafana dashboard was empty
  • Introduced: Update Kubernetes from v1.15.3 to v1.16.0 #543

* Prometheus was configured to use kubernetes discovery
of etcd targets based on nodes matching the node label
node-role.kubernetes.io/controller=true
* Kubernetes v1.16 stopped permitting node role labels
node-role.kubernetes.io/* so Typhoon renamed these labels
(no longer any association with roles) to
node.kubermetes.io/controller=true
* As a result, Prometheus didn't discover etcd targets,
etcd metrics were missing, etcd alerts were ineffective,
and the etcd Grafana dashboard was empty
* Introduced: #543
@dghubble dghubble force-pushed the fix-etcd-metrics-scrape branch from 0ccc68b to 19de38b Compare October 4, 2019 02:07
@dghubble dghubble changed the title Fix Prometheus etcd metrics scraping Fix Prometheus etcd target discovery Oct 4, 2019
@dghubble dghubble merged commit 19de38b into master Oct 6, 2019
@dghubble dghubble deleted the fix-etcd-metrics-scrape branch October 6, 2019 01:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant