error when getting metric values error when getting scalers ScaledObject.keda.sh "" not found

[boylee1111]

Report

error when getting metric values error when getting scalers ScaledObject.keda.sh "" not found while retrieving external metrics using kubectl get --raw ...

Expected Behavior

Successfully retrieve the external metrics.

Actual Behavior

Error from server: rpc error: code = Unknown desc = error when getting metric values error when getting scalers ScaledObject.keda.sh "" not found

Steps to Reproduce the Problem

1. Install Keda 2.9.1: kubectl apply -f https://github.com/kedacore/keda/releases/download/v2.9.1/keda-2.9.1.yaml
2. Follow https://github.com/kedacore/sample-go-rabbitmq#rabbitmq-consumer-and-sender to install RabbitMQ and publish message.
3. I was able to get metrics using

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1"

{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"external.metrics.k8s.io/v1beta1","resources":[{"name":"s0-rabbitmq-hello","singularName":"","namespaced":true,"kind":"ExternalMetricValueList","verbs":["get"]}]}

However, it does not work when I provided metrics name s0-rabbitmq-hello:

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/s0-rabbitmq-hello"

Error from server: rpc error: code = Unknown desc = error when getting metric values error when getting scalers ScaledObject.keda.sh "" not found

It hits the same error when I use https://github.com/kedacore/http-add-on

Logs from KEDA operator

From the logs, somehow it attempts to get ScaledObject with empty name:

2023-01-10T21:38:25Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"rabbitmq-consumer","namespace":"default"}, "namespace": "default", "name": "rabbitmq-consumer", "reconcileID": "545a5324-de5f-4987-9dfe-bb3a24e5ad0c"}
2023-01-10T21:59:24Z	ERROR	scalehandler	failed to get ScaledObject	{"name": "", "namespace": "default", "error": "ScaledObject.keda.sh \"\" not found"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
	/workspace/pkg/scaling/scale_handler.go:240
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScalersCacheForScaledObject
	/workspace/pkg/scaling/scale_handler.go:198
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScaledObjectMetrics
	/workspace/pkg/scaling/scale_handler.go:389
github.com/kedacore/keda/v2/pkg/metricsservice.(*GrpcServer).GetMetrics
	/workspace/pkg/metricsservice/server.go:45
github.com/kedacore/keda/v2/pkg/metricsservice/api._MetricsService_GetMetrics_Handler
	/workspace/pkg/metricsservice/api/metrics_grpc.pb.go:79
google.golang.org/grpc.(*Server).processUnaryRPC
	/workspace/vendor/google.golang.org/grpc/server.go:1340
google.golang.org/grpc.(*Server).handleStream
	/workspace/vendor/google.golang.org/grpc/server.go:1713
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/workspace/vendor/google.golang.org/grpc/server.go:965
2023-01-10T22:03:49Z	ERROR	scalehandler	failed to get ScaledObject	{"name": "", "namespace": "default", "error": "ScaledObject.keda.sh \"\" not found"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
	/workspace/pkg/scaling/scale_handler.go:240
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScalersCacheForScaledObject
	/workspace/pkg/scaling/scale_handler.go:198
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScaledObjectMetrics
	/workspace/pkg/scaling/scale_handler.go:389
github.com/kedacore/keda/v2/pkg/metricsservice.(*GrpcServer).GetMetrics
	/workspace/pkg/metricsservice/server.go:45
github.com/kedacore/keda/v2/pkg/metricsservice/api._MetricsService_GetMetrics_Handler
	/workspace/pkg/metricsservice/api/metrics_grpc.pb.go:79
google.golang.org/grpc.(*Server).processUnaryRPC
	/workspace/vendor/google.golang.org/grpc/server.go:1340
google.golang.org/grpc.(*Server).handleStream
	/workspace/vendor/google.golang.org/grpc/server.go:1713
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/workspace/vendor/google.golang.org/grpc/server.go:965

KEDA Version

2.9.1

Kubernetes Version

1.26

Platform

Google Cloud

Scaler Details

No response

Anything else?

No response

[aberres]

I am running into this as well.

[JorTurFer]

Try using the full query string including the scaledobject. It seems like the scaledobjects is not generated on the fly if in comes empty
https://keda.sh/docs/2.9/operate/metrics-server/#how-to-get-metric-when-multiple-scaledobjects-have-the-same-metric-name

[aberres]

@JorTurFer Thanks, this worked. But I have to after looking at the docs again I do not understand why this is needed.

[JorTurFer]

Most probably because after the changes in the architecture, something was broken, we should review it xD

[zroubalik]

This doesnt work anymore:

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/sample-ns/s1-rabbitmq-queueName2"

A selector with ScaledObject's name needs to be specified:

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/sample-ns/s1-rabbitmq-queueName2?labelSelector=scaledobject.keda.sh%2Fname%3D{ScaledObjectName}"

[rh-voice]

I am still getting this error ERROR scalehandler failed to get ScaledObject {"name": "", "namespace": "default", "error": "ScaledObject.keda.sh \"\" not found"}

and I dont have any scaledobjects deployed; its just a fresh Keda installation.

[JorTurFer]

The commit hasn't been released yet, but it's already in main. Next minor release will solve it

[rh-voice]

Thanks, it doesn't seem to be disruptive to anything else; i deployed a test ScaledObject and it worked fine.

so no harm in using this version for now, right ?

[JorTurFer]

You are right, that's just a warning produced by a call from api server listing the metrics. Apart from the annoying message, you are safe