Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prometheus scraper misconfigured in the k8s service #136

Open
frdeboffles opened this issue Jan 21, 2022 · 0 comments · May be fixed by #137
Open

Prometheus scraper misconfigured in the k8s service #136

frdeboffles opened this issue Jan 21, 2022 · 0 comments · May be fixed by #137

Comments

@frdeboffles
Copy link

What happened:
Getting plenty of errors like:

2022/01/21 00:39:53 http: TLS handshake error from 10.97.36.76:42820: remote error: tls: bad certificate

In my cluster, this IP relates to Prometheus server. Looking at the service configuration I found that the Prometheus scraper annotations were pointing at the webhook TLS port instead of the metrics port.

What you expected to happen:
No errors in the logs

How to reproduce it (as minimally and precisely as possible):
Install Prometheus in the cluster where the webhook is running with scrape enabled for service endpoints.

Anything else we need to know?:

Environment:

  • AWS Region: us-east-1
  • EKS Platform version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.platformVersion): eks4
  • Kubernetes version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.version): 1.21
  • Webhook Version: master branch at a65cc3d9c61cf6fc43f0f985818c474e0867d786
frdeboffles pushed a commit to frdeboffles/amazon-eks-pod-identity-webhook that referenced this issue Jan 21, 2022
@vai-frd
Exposed the metrics port in the k8s service and configured Prometheus…
264b33c 
… annotations to use the metrics port instead of the webhook one

Fix: aws#136
@frdeboffles frdeboffles linked a pull request Jan 21, 2022 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix Prometheus scraper config in k8s service frdeboffles/amazon-eks-pod-identity-webhook
1 participant
@frdeboffles