Azure Storage Queue scaler not working when using Azure Managed Identities

[glzbcrt]

Expected Behavior

Use Azure Storage Queue scaler using Azure Managed Identity.

Actual Behavior

The scaler is not able to authenticate using Azure Managed Identity because the Metadata HTTP header is not sent to Azure IMDS service.
The code where the request is created and sent can be found at the address below.

keda/pkg/scalers/azure/azure_aad_podidentity.go

Line 23 in d83a747

req, err := http.NewRequest("GET", urlStr, nil)

Steps to Reproduce the Problem

1. Deploy an AKS cluster
2. Deploy KEDA
3. Create a TriggerAuthentication to use Azure managed identities
4. Create a ScaledObject to use the authentication from step 3 and to scale based on an Azure Storage Queue.

Logs from KEDA operator

This is the log from the KEDA operator showing the authentication error.

2021-02-04T12:06:23.510Z        ERROR   azure_queue_scaler      error)  {"error": "-> github.com/Azure/azure-storage-queue-go/azqueue.newStorageError, /go/pkg/mod/github.com/!azure/azure-storage-queue-go@v0.0.0-20191125232315-636801874cdd/azqueue/zc_storage_error.go:42\n===== RESPONSE ERROR (ServiceCode=InvalidAuthenticationInfo) =====\nDescription=Authentication information is not given in the correct format. Check the value of Authorization header.\nRequestId:821db1a7-9003-000f-76ee-fac0dd000000\nTime:2021-02-04T12:06:23.5090768Z, Details: \n   Code: InvalidAuthenticationInfo\n   GET https://REDACTED.queue.core.windows.net/myqueue?comp=metadata&timeout=61\n   Authorization: REDACTED\n   User-Agent: [Azure-Storage/0.3 (go1.15.6; linux)]\n   X-Ms-Client-Request-Id: [097ec2dd-4cbf-443f-5405-21ad9b5b925b]\n   X-Ms-Version: [2018-03-28]\n   --------------------------------------------------------------------------------\n   RESPONSE Status: 400 Authentication information is not given in the correct format. Check the value of Authorization header.\n   Content-Length: [297]\n   Content-Type: [application/xml]\n   Date: [Thu, 04 Feb 2021 12:06:23 GMT]\n   Server: [Microsoft-HTTPAPI/2.0]\n   X-Ms-Error-Code: [InvalidAuthenticationInfo]\n   X-Ms-Request-Id: [821db1a7-9003-000f-76ee-fac0dd000000]\n\n\n"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.3.0/zapr.go:132
github.com/kedacore/keda/v2/pkg/scalers.(*azureQueueScaler).IsActive
        /workspace/pkg/scalers/azure_queue_scaler.go:126
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScaledObjectScalers
        /workspace/pkg/scaling/scale_handler.go:206
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
        /workspace/pkg/scaling/scale_handler.go:195
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
        /workspace/pkg/scaling/scale_handler.go:138

This is the log from the node-managed identity (NMI) pod showing that an HTTP request was received without the Metadata header and the return status was 400.

E0204 11:11:55.379054 1 server.go:322] metadata header is not specified, req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=10.0.0.15
I0204 11:11:55.379098 1 server.go:192] status (400) took 56203 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=10.0.0.15

Specifications

• KEDA Version: 2.1.0
• Platform & Version: Azure Kubernetes Service
• Kubernetes Version: 1.19.3
• Scaler(s): Azure Storage Queue

[tomkerkhove]

@ahmelsayed I don't think we even support Managed Identity as a whole, no? At least it's not documented on https://keda.sh/docs/2.1/scalers/azure-storage-queue/.

[ahmelsayed]

@tomkerkhove I think so, I assumed this meant podIdentity mentioned in the example of the TriggerAuthentication there, but I'm not sure if there are other Manged Identity features that we could consider?

[tomkerkhove]

Ha, we should document this then as it's not on https://keda.sh/docs/2.1/scalers/azure-storage-queue/

[tomkerkhove]

I'll pick it up

[tomkerkhove]

It is documented, I'm just blind 😂