How to deploy multiple Prometheus-adapters in the same k8s cluster？

[UrmsOne]

Hi. I would like to ask whether it is supported to deploy multiple Prometheus-adapters in the same cluster.

In my cluster, the knative has defined an apiservice named custom.metrics.k8s.io/v1beta1 for HPA.

root@ubuntu:/workspace/gospace# kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 |jq
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "revisions.serving.knative.dev/concurrency",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "revisions.serving.knative.dev/rps",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

I realized GPU metrics collection through prometheus-exporter, and now I want to deploy a Prometheus-adapter to achieve HPA

for GPU idle rate. How should I deal with related applications？I could make sure knative applications and

Prometheus-adapter can run normally.

I try to modify the groupversion to v1beta1.gpu.custom.metrics.k8s.io, but it doesn't work properly.

apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.gpu.custom.metrics.k8s.io
spec:
  service:
    name: custom-metrics-apiserver
    namespace: custom-metrics
  group: gpu.custom.metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100

Thanks for helping me to solve this.

[dgrisonnet]

Hi @UrmsOne,

If you want to use the custom metrics API, you can only do so via the custom.metrics.k8s.io aggregated API, you cannot extend it in any way. Also, you can only bind one aggregator per apiservice, which means that if you are using one prometheus-adapter instance that serve this endpoint, no other instance will be able to serve it. So to answer your original question, it is not possible to deploy multiple prometheus-adapter instances for your particular use case as only one will be able to serve the custom metrics API.

Why do you need 2 instances in the first place? Couldn't you achieve the same result with only one prometheus-adapter instance serving both your knative metrics and your gpu metrics? You could then select the metrics you want to autoscale on directly in your HPA configurations.

[UrmsOne]

@dgrisonnet Thanks for your reply.

My scenario is that two components use the same apiservice. One is the knative application deployed by kubeflow, and the other is Prometheus-adapter.

The knative application already exists in my cluster, which is occupies custom.metrics.k8s.io to achieve HPA. Now I want to deploy a Prometheus-adapter to collect GPU metrics and implement HPA through the GPU metrics. What should I do to make knative application and Prometheus-adapter not affect each other.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[dgrisonnet]

@UrmsOne you can't do that yet, but this is being investigated in kubernetes-sigs/custom-metrics-apiserver#70.

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.