-
Notifications
You must be signed in to change notification settings - Fork 1k
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to fetch metrics from external metrics API: Internal error occurred: DatadogMetric is invalid #4415
Comments
Hi, This means that both systems can't work together and this could be the reason behind the behavior you are describing |
Yeah, most likely it is this problem. |
The output of that command is When we were setting up KEDA, we did get an error about the metrics server so we added:
to remove the duplicate metrics server |
The problem is that even though KEDA operator is setting the HPA correctly, the HPA controller can't get the value for scaling because the endpoint is mapped to You cannot use KEDA and datadog metrics server at the same time |
Oh interesting. Is there a recommended path forward? Does it make more sense to use just autoscaling with Datadog queries instead? What is the use case for the Datadog scaler with KEDA if you can't use the metrics server? We were following this blogpost to implement the Datadog Scaler for KEDA which didn't mention any issues with the metrics server and we couldn't find other documentation mentioning not being able to use the datadog metrics server while using KEDA. |
Do you mean a migration path? We don't have any migration path sorry
Datadog scaler uses the Datadog SDK to get the metrics directly from the Datadog API, that's why the Datadog metrics server isn't necessary as KEDA does that job of requesting the metrics from Datadog Api You can find more information about the limitation in the upstream issue: kubernetes-sigs/custom-metrics-apiserver#70 |
Converting to discussion |
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
Report
When deploying KEDA using a Datadog Scaler, the HPA created by KEDA returns
ScalingActive False FailedGetExternalMetric the HPA was unable to compute the replica count: unable to get external metric infra-integ/s0-datadog-avg-nginx-net-request_per_s/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: dummy-app-flagger,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: Internal error occurred: DatadogMetric is invalid, err: Global error (all queries) from backend
and the target value is always<unknown>
kubectl describe hpa keda-hpa-dummy-app-flagger
kubectl describe datadogmetric dcaautogen-56cd4bb9dd5640062b98461f22b0da37904cf7
The scaled object:
The datadog cluster agent is configured to use the datadog metrics provider:
We have tried using KEDA v2.7.0, 2.6.1, and 2.6.0 and see the same error with each version.
When following the Datadog docs to create an HPA, we are able to get metrics and see scaling
Expected Behavior
Be able to get metrics returned from Datadog and scale appropriately
Actual Behavior
Error parsing the metric name and therefore unable to return a metric value to the HPA
Steps to Reproduce the Problem
kubectl describe hpa keda-hpa-dummy-app-flagger
andkubectl describe datadogmetric dcaautogen-56cd4bb9dd5640062b98461f22b0da37904cf7
Logs from KEDA operator
Log from cluster agent
KEDA 2.10.0:
When trying KEDA 2.7 and without a
DatadogMetric
object by setting:There are no errors in the operator:
KEDA Version
2.10.0
Kubernetes Version
1.24
Platform
Amazon Web Services
Scaler Details
Datadog
Anything else?
We've tried adjusting the
age
and themetricUnavailableValue
as well and have had no luck.We have tried using KEDA v2.7.0, 2.6.1, and 2.6.0 and see the same error with each version.
The text was updated successfully, but these errors were encountered: