Running together with metrics-server

[igoratencompass]

What happened?:

After deploying prometheus-adapter In a cluster where I have metrics-server running I get a clash around the APIService/v1beta1.metrics.k8s.io resource as it seems being created by both. Do we need to remove metrics-server before installing prometheus-adapter ?

Environment:

• prometheus-adapter version: helm chart v4.1.1
• prometheus version: helm chart v15.9.0
• Kubernetes version (use kubectl version): EKS v1.22
• Cloud provider or hardware configuration: AWS
• Other info:

[shinebayar-g]

I couldn't find the source for this. But IIRC I don't think you're supposed to run both. You can remove metrics-server and keep using prometheus-adapter as it covers the functionality of the metrics-server. Pls correct me if I'm wrong.

[igoratencompass]

I understand it same, the adapter should provide the functionality of the metrics-server plus custom and external metrics support. It sucks that kubectl top does not work upon metrics-server removal as it is an easy way to confirm if everything is a OK.

To be even more complicated we also use Datadog with external metrics which in turn clashes about APIService/v1beta1.external.metrics.k8s.io resource with the adapter, seems no way to win this three-way battle and have all corners covered :-)

[shinebayar-g]

It sucks that kubectl top does not work upon metrics-server removal as it is an easy way to confirm if everything is a OK.

It should work? (If prometheus-adapter is installed afterwards).

[logicalhan]

/triage accepted
/assign @dgrisonnet

[dgrisonnet]

It should work? (If prometheus-adapter is installed afterwards).

Yeah if you have configured resource metrics, it should work.

To be even more complicated we also use Datadog with external metrics which in turn clashes about APIService/v1beta1.external.metrics.k8s.io resource with the adapter, seems no way to win this three-way battle and have all corners covered :-)

There is an effort that was started a while ago but didn't make much progress, to support multiple external metrics providers: kubernetes-sigs/custom-metrics-apiserver#70.

But if you use both Prometheus and Datadog metrics for autoscaling I would recommand KEDA: https://github.com/kedacore/keda. It is a CNCF project that support autoscaling on both datasources and many more.

[dgrisonnet]

/kind support

[igoratencompass]

It should work? (If prometheus-adapter is installed afterwards).

The problem is we use ArgoCD that constantly reconciles the deployments hence if we have datadog, metrics-server and prometheus-adapter installed they are constantly overwriting each others shared objects, quite unnerving watching this going on for ever in the UI really.

[k8s-triage-robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

[dashpole]

Sorry we didn't get to this. If this is still affecting you, feel free to reopen.
/close

[k8s-ci-robot]

@dashpole: Closing this issue.

In response to this:

Sorry we didn't get to this. If this is still affecting you, feel free to reopen.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[satdeveloping]

@dashpole This is still a thing, can you reopen?

[dgrisonnet]

I feel like this should remain closed as I have already shared the current initiative that would fix your issue and a potential alternative you could use in the meantime. There is nothing more that can be done.

As I mentioned before there are three ways to go about the problem:

1. Use dedicated APIs for each metrics server. There are three of them and you have three metrics server so maybe something can be worked there. This is quite hacky and might not be suited to what you are trying to achieve
2. Move to using a combination of KEDA + metrics-server as KEDA could be a good replacement to datadog and prometheus-adapter as it supports both datadog and prometheus as scalers.
3. Wait for Proposal: Support multiple custom/external metrics servers in the cluster custom-metrics-apiserver#70 to be implemented