-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
keda-operator-metrics-apiserver restarts frequently #2058
Comments
Now i saw this error on the keda metric server log.
Even for /readyz , i saw a similar webhook call failed error. This happens when multiple scalars related metrics start to report value , meaning surge in SQS available message count & scaling needs to happen. Additional log :
|
@rangarb885 to answer your questions first:
not currently, but iirc the operator could because it elects a leader. I'll defer to @zroubalik and @tomkerkhove whether they would want to take advantage of that on to the problem you filed this issue for, I'm not as familiar with the metrics server code as others, but my (slightly educated) hunch is that the max in-flight limit filter is to blame (see here). still working on tracing the metrics server down to that point, but it is in the stack trace you pasted. perhaps others could chime in here. |
You should be able to scale the metrics server as it uses a regular kubernetes service for loadbalancing The default |
Thank you @arschles & @ahmelsayed . |
@ahmelsayed Good to know. is there any technical requirement that the metrics server is a singleton? if not, does it make sense to update the helm chart to allow for N>1 replicas? |
If it's possible having more than one instance of metrics server, I think that we should explain it explicitly because it's important to maintain the HA, I thought that 1 instance is a technical requirement :( |
I knew that I read something about metric server limitation, and I did in KEDA docs xD |
Yes, it's documented in our HA docs indeed: https://keda.sh/docs/latest/operate/cluster/#high-availability |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions. |
This issue has been automatically closed due to inactivity. |
Report
keda-operator-metrics-apiserver gets restarted frequently when there are 6 scalar (SQS). Metric is polled every 30s from SQS. This cause metric report missing for the window of time when it restarts.
Expected Behavior
keda-operator-metrics-apiserver needs to be stable
Actual Behavior
keda-operator-metrics-apiserver restarts frequently.
Steps to Reproduce the Problem
Logs from KEDA operator
Strangely there is no log on the metric server when it restarts.
Below is the log from HPA which clearly shows the metric server has some issue.
Also during that time when i try to fetch the metric using kubectl, i received
Unable to connect to the server: unexpected EOF
KEDA Version
2.4.0
Kubernetes Version
1.21
Platform
Amazon Web Services
Scaler Details
SQS
Anything else?
General questions:
The text was updated successfully, but these errors were encountered: