-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
KEDA might break existing deployment on cluster which already has another External Metrics Adapter installed #470
Comments
this should probably have 'needs-discussion' label, I am not able to assign it. |
@zroubalik I’ll make sure you get label permissions :). Adding needs-discussion and help-wanted in case someone gets change to validate if it is a bug |
@jeffhollan I am pretty sure this is by design right now. I will take a look at what options are available to chain metric servers. |
@Aarthisk I am planning to look at other options as well. |
This is a known limitation of the custom/external metrics API. A possible solution is to come up with an aggregation API as per kubernetes-sigs/custom-metrics-apiserver#3. Knative's HPA support suffers from the same limitation. |
@markusthoemmes thanks for the info |
What's the status here, not sure if we can do anything about this? |
We should keep it open until it gets resolved in https://github.com/kubernetes-sigs/custom-metrics-apiserver/ |
Ran into this because datadog helm chart also creates the APIService object with the same name of |
We ran into the same situation as @v-yarotsky mentioned. We have datadog installed with datadog-cluster-agent-metrics-api:
It does not overwrite or break existing
|
That issue might be related to kedacore/charts#88 |
@tomkerkhove unfortunately it is not. Looking at that issue and the last error message from the screenshot, it is missing some helm labels, so the validation is failing. |
I did naively try to remove the
I did see you @zroubalik have this issue: kubernetes-sigs/custom-metrics-apiserver#70 proposed. If you have a branch/fork with this fix, we are more than happy to try it out. We are also happy to help with implementation. |
@hinling-sonder it is still just a proposal, I should have start working on this in a very near future. But to get it working, a change on Datadog side will be needed as well. |
hello all, we are also experiencing the same issue here, we are using |
We are definately aware of this and that it's a pain point, sorry! We have a very smart guy in our team who will look into a POC for contributing this upstream. |
The option that you could choose is using KEDA with Datadog Scaler |
trying to deploy keda on a k8s cluster with existing datadog deployment :
and same error when trying to deploy datadog on a cluster with existing keda deployment:
is there a workaround for this? |
@alxgruU as mentioned above, you cannot have both KEDA and another service that is using |
Team I'm facing the below error. What is workaround? |
hi @slv-306 |
We have another metrics-server in place. Is there any workaround. By default we have |
if the metrics server that you already have uses |
Is there any way to push metrics to stackdriver with Keda |
No, unfortunately not. |
Can we use two triggers for autoscaling single deployment. Like one trigger for CPU while other for RPS |
do you mean using KEDA? |
@slv-306 As @JorTurFer mentioned this is supported and documented in our FAQ: https://keda.sh/docs/latest/faq/ May I ask you to create a GitHub Discussion for asking questions that are not related to this issue please? This helps us keep the conversation more focused - Thank you! |
FAQ update: kedacore/keda-docs#950 |
Hello guys, I managed to workaround this disabling the metricsProvider from Datadog. In this way you can deploy both charts in your cluster. Keep in mind that the Datadog metricProvider is for Datadog to autoscale using Custom Metrics. https://docs.datadoghq.com/containers/guide/cluster_agent_autoscaling_metrics/ |
We are working on a proposal to fix this limitation directly in k8s, but we are still drafting the KEP. I hope that during next months it can be fixed, but currently, thanks for your workaround! |
Hi guys! Just for those who still struggle to fix this issue (integrating Datadog with Keda) I finally made it to install datadog in my GKE with You may not need metricsProvider to be enabled as it's only used for datadog auto-scale. |
About this issue, we are working on a KEP to support multiple metrics servers natively in Kubernetes 😋 |
This might be a good candidate for documentation, Troubleshooting/FAQ section. @JorTurFer WDYT? |
It's already documented: https://keda.sh/docs/2.12/faq/#kubernetes |
I meant the coexistence with Datadog as mentioned in the quoted #470 (comment) |
ah, okey. It could be, but I'm not 100% sure if we haven't written it yet |
Any updates here? It has been almost 4 years. I encountered this issue (APIService already exists) in my GKE with Google-Managed Prometheus and custom-metric-stackdriver.
|
We are still blocked by Kubernetes upstream |
KEDA is using metrics adapter based on custom-metrics-apiserver
library. As part of the deployment, user need to specify cluster wide
APIService
object namedv1beta1.external.metrics.k8s.io
, see in the library example and in KEDA deployment .I wonder what would happen, if user has already deployed another Metrics Adapter (which is using the same
APIService
based approach) and we try to install Keda. It will probably replace the originalAPIService
definition with KEDA one, so KEDA will work, but the original stuff installed on cluster probably not. We should not break things or should make clear, that this could happen.We should investigate what are the possibilities and whether there are a better solutions on how to deal with the metrics. Or my assumptions are wrong, so please correct me in this case.
The text was updated successfully, but these errors were encountered: