Provide configuration option for counting invisible/visible messages in Azure Storage Queue scaler

[tomkerkhove]

Proposal

I believe we should introduce a new configuration option that will influence this:

  metadata:
    queueName: orders
    queueLength: '5'
+   queueLengthStrategy: Default|VisibleOnly|InvisibleOnly
    activationQueueLength: '50'
    connectionFromEnv: STORAGE_CONNECTIONSTRING_ENV_NAME
    accountName: storage-account-name
    cloud: AzureUSGovernmentCloud

Use-Case

We've been going back and forward if Azure Storage Queue scaler should consider invisible during evaluation or not, but this is not ideal for everyone. Example #4432.

Is this a feature you are interested in implementing yourself?

No

Anything else?

No response

[spewu]

This would be perfect for us. For some specific queues, we leverage the visibility of messages in the queue, to postpone processing them. The current implementation means that KEDA will keep scaling up, because there are invisible messages in the queue, but the jobs it creates have nothing to do, since there are no visible messages in the queue.

So, being able to specify that for this particular trigger, we only want to consider the visible messages, would allow this to work for us without continuously creating useless jobs in the cluster.

[dgancho]

Hi @tomkerkhove et al, is there any known solution/workaround here or a path forward? The issue still persists and blocks from using Azure Container App Jobs as advertised.

[tomkerkhove]

The PR is open as you can see above your comment.

Azure Container App Jobs as advertised

Mind elaborating on this? =)

[rubenmamo]

The PR is open as you can see above your comment.

Azure Container App Jobs as advertised

Mind elaborating on this? =)

@tomkerkhove I have an Azure Container App Job set to listen to an Azure Queue. I have the scaler to poll every minute, however, when I create a queue item for 10 minutes in the future, if I have only this item in the queue, the job gets started every minute even though the queue item is not visible yet, when the 10 minutes passes then it executes the job with the queue.

If we could ignore invisible messages, then Azure Container App Jobs will only get triggered if there is a visible message when polled.