Ability to delay scaling after deployment changes

[leonfibal]

Hi,

We are facing some issues/problems with integrating our applications with keda.

Background:

We have .net based applications with MassTransit/RabbitMQ. When an instance of application starts it configures exchanges and queues (adds proper exchanges mostly).

Example:

The application X is deployed on kubernetes (with keda scaled object). Keda scales the application within 0-3 range. The application listens to event A.

One day we changes the application and now it starts to listen to event B. We deploy the application on kubernetes cluster. Number of pods is 0. What happens? RabbitMQ queues and exchanges are still configured as it was before our changes. Event B never reaches our application, because there are only exchanges for event A (which will never come again for some reason).

No scaling occurs, no pod with newer images starts.

What can we do with that?

We thought about an ability to delay scaling after the deployment changed - maybe some kind of readiness check for keda? Or simple interval of time to await.

Or maybe the solution is ready 😃 and you can help us without code changes.

[zroubalik]

Hi, sorry I don't follow what is your problem. Could you please give us more details?

One day we changes the application and now it starts to listen to event B. We deploy the application on kubernetes cluster. Number of pods is 0. What happens? RabbitMQ queues and exchanges are still configured as it was before our changes.

RabbitMQ queues defined where?

[leonfibal]

There is one queue per application, it changes rarely. The problem is with exchanges, because they configured when the application starts (basing on registered implementations of IConsumer). Example: https://masstransit-project.com/usage/consumers.html#consumer.

[zroubalik]

I still don't get it sorry 🤷‍♂️

So my understanding is:

1. You have Deployment that's being scaled by RabbitMQ, this points to the queue A (the same queue is defined in the ScaledObject).
2. You change the Deployment to consume from queue B (and change the ScaledObject to point at queue B as well)
3. No scaling is happening... --> here I don't understand why and why it is related to KEDA.

[leonfibal]

Queue can be bound to many exchanges so an application can consume many types of events from one queue.

1. I have Deployment that's being scaled by RabbitMQ, this points to the queue A (the same queue is defined in the ScaledObject).
2. I change the deployment (the image version in fact) to consume more types of events (lets say - one more event). The deployment and ScaledObject point to queue A.
3. No scaling is happening

There is missing configuration step in point 3, because configuration of exchanges bound to the queue is done in runtime. There is no runtime, because keda scaled deployment to 0.

The problem is with external dependencies that are configured in runtime by the application, but it cannot occur because the deployment has 0 pods. KEDA maintains 0 pods, but do not know that metric value could be higher if the application had a chance to configure those external dependencies.

[tomkerkhove]

Maybe KEDA is not able to correctly decide if it should scale due to misconfiguration, there are a few options:

1. Change minimum replica count to always have 1 instance
2. Check the logs of KEDA to ensure there are no issues with the configuration

[zroubalik]

okay, gotcha. I am still not sure what could KEDA do about this as this is something very specific.

You can temporarily remove the ScaledObject, so the Deployment will be scaled to default number of replicas, let it do the configuration changes and the re-create SO, or you can set the minReplicaCount to 1 or in the future you can use this potential feature: #944 (adding a particular label on ScaledObject, which will mean that this SO is then ignored by KEDA and scaled to 0 or 1 or any arbitrary number of replicas.)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.