cortex header is missing

[aberenshtein]

I'm trying to test it with keda 2.7.0
This is my keda scaledObject

triggers:
- metadata:
cortexOrgId: saas-stg
metricName: bull_queue_houzz1_waiting
query: >-
bull_queue_houzz1_delayed{queue="email"} +
bull_queue_houzz1_waiting{queue="email"}
serverAddress: http://<our cortex server>/prometheus
threshold: '1'
type: prometheus
And I get this error

2022-05-09T11:02:59+03:00 1.6520833799251316e+09 ERROR scalehandler Error getting scale decision {"scaledobject.Name": "ivy-worker-email", "scaledObject.Namespace": "houzz1", "scaleTarget.Name": "ivy-worker-email", "error": "prometheus query api returned error. status: 401 response: no org id\n"}
If I make a direct query to http://<our cortex server>/prometheus?query=<query from above>
and I add the X-Scope-OrgID header I get a 200 response

If I remove the header, I'm able to reproduce the issue from above.

What am I doing wrong? It seems the header is not added properly

[xoanmm]

I would like to tackle this 😄

[zroubalik]

@xoanmm awesome, if you are able to tackle this soon, we might want to include in 2.7.1 release which is about to happen.

[aberenshtein]

@xoanmm do you know what's the problem?
Maybe I can fix it and build the dockerfile locally?

[zroubalik]

@aberenshtein you can try to build docker images locally with the fix you have proposed in the discussion. Also don't forget to rebuild both Operator and Metrics Server images.

You can do it this way once you do the changes in the code, you can run following command, which will build & push images to dockerhub and then deploy this version to your k8s cluster (please remove existing KEDA installation from the cluster):

IMAGE_REPO=your_docker_username VERSION=promFix make publish && IMAGE_REPO=your_docker_username VERSION=promFix make deploy

[aberenshtein]

Cool. Thanks, @zroubalik
I think I'll wait for @xoanmm

[xoanmm]

I think that the problem is related with what you mention in the issue
Maybe you can try changing the file https://github.com/kedacore/keda/blob/main/pkg/scalers/prometheus_scaler.go#L197 adding the following:

if s.metadata.cortexOrgID != nil && s.metadata.cortexOrgID  !"" {
	req.Header.Add(promCortexHeaderKey, s.metadata.cortexOrgID)
}

[aberenshtein]

I found the problem...
It's just a typo. cortexOrgId instead of cortexOrgID

[zroubalik]

@aberenshtein to clarify, a typo in your ScaledObject?

[aberenshtein]

yep, in mine

[djetelina]

That's a typo in documentation: https://keda.sh/docs/2.8/scalers/prometheus/