Support for X-Scope-OrgID header in Prometheus scaler

[romanvog]

Proposal

Need to add support for X-Scope-OrgID header as part of the query to a multi tenant cortex (prometheus metrics), since it's not possible to get a response without it when multi tenancy is enabled.

Use-Case

We're using scalers from type prometheus, and recently moved to a multi tenant cortex to serve metrics from multiple different prometheus instances in different clusters. To support multi tenancy, we need to provide X-Scope-OrgID header for cortex to know which prometheus data to return.
We need to pass this header in keda's prometheus scaler as well if we want to use cortex endpoint (which we do).

Anything else?

No response

[JorTurFer]

Hi @romanvog
This sounds totally reasonable 😄
Are you willing to contribute?

[romanvog]

hey @JorTurFer,
not familiar with go, but i can give it a try :)
want to point me to the place in code where prometheus type is implemented?

[JorTurFer]

Hey!
This is the scaler code

[xoanmm]

I would like to tackle this 😄

[JorTurFer]

Thanks a lot! ❤️

[aberenshtein]

@JorTurFer @xoanmm
I'm trying to test it with keda 2.7.0
This is my keda scaledObject

 triggers:
    - metadata:
        cortexOrgId: saas-stg
        metricName: bull_queue_houzz1_waiting
        query: >-
          bull_queue_houzz1_delayed{queue="email"} +
          bull_queue_houzz1_waiting{queue="email"}
        serverAddress: http://<our cortex server>/prometheus
        threshold: '1'
      type: prometheus

And I get this error

2022-05-09T11:02:59+03:00 1.6520833799251316e+09	ERROR	scalehandler	Error getting scale decision	{"scaledobject.Name": "ivy-worker-email", "scaledObject.Namespace": "houzz1", "scaleTarget.Name": "ivy-worker-email", "error": "prometheus query api returned error. status: 401 response: no org id\n"}

If I make a direct query to http://<our cortex server>/prometheus?query=<query from above>
and I add the X-Scope-OrgID header I get a 200 response

If I remove the header, I'm able to reproduce the issue from above.

What am I doing wrong? It seems the header is not added properly

[aberenshtein]

https://github.com/kedacore/keda/blob/main/pkg/scalers/prometheus_scaler.go#L197
Maybe this should be

if s.metadata.cortexOrgID != nil {
		req.Header.Add(promCortexHeaderKey, s.metadata.cortexOrgID)
	}

[JorTurFer]

Hey,
Could you open another issue with this bug? Just to track it better 🙏

[aberenshtein]

Sure. #3024
Can you add a bug label to my issue please