Editing ScaledObject on Workload in Production

[corradomatt]

Hey all! First, thanks for a great project while I'm not using it in production yet testing is going very well and I'm excited to see it in action.

I've been running load tests using Keda ScaledObjects to control the HPA for my Deployment. I'm using the gcp-stackdriver trigger on a custom GCP log metric from the workload itself and it's been working great. However, I wanted to test the process for making adjustments to the metadata.targetValue of my trigger and noticed something concerning. I have the minReplicaCount currently set to 10 and during a load test when I adjusted the targetValue I noticed that my workload was scaled down from 170 pods to 10 before the new calculation kicked in a scaled it up to the expected 200+ pods. The process happened quickly enough and my load test hardly noticed the issue, but I know this wouldn't be the case in production.

How can I ensure that the HPA doesn't scale down my workload to the minReplicaCount value when making minor adjustments to the ScaleObject?

[corradomatt]

I think that I see the issue in the keda-operator logs...

{
  "error": "rpc error: code = Canceled desc = grpc: the client connection is closing",
  "level": "error", "logger":"gcp_stackdriver_scaler",
  "msg":"error getting metric value",
  ...
}

{
  "error": "scaler with id 0 not found, len = 0, cache has been probably already invalidated",
  "level": "error",
  "logger": "scale_handler",
  "msg": "error getting scale decision",
  ...
}

It looks like the connection to GCP metrics breaks and then the scaler uses the fallback.replicas number. Here's what I've set for the fallback values...

fallback:
    failureThreshold:      3
    replicas:               10

Should I increase the failureThreshold value maybe? Or do I just need to adjust the fallback.replicas to match what is currently running when I make any adjustment?

[rickbrouwer]

I have created the following issue: #6450.

I have basically finished the code and tested it on our own cluster. The only thing is that I still have to make tests, which will cost me a lot of time. (Yes, I know, it is wrong to first make code and then the tests, but I wanted to see it with my own eyes first ;) )

[rickbrouwer]

I have created the following PR: #6464.

[JorTurFer]

Hello
Sorry because I'm so late in the discussion :(
I think that

fallback:
    failureThreshold: 3
    replicas: 10
    useCurrentReplicas: true

is actually almost like not setting the fallback. I mean, in the scenario that you have described (having 170 pods and fallback with 10), not having the fallback will just keep your 170 replicas untouched. This feature can be a benefit in the other direction (having 3 instances and fallback to 10).

When KEDA applies the fallback, it shows the fact clearly in the logs, have you seen a log in KEDA operator saying that the fallback has been done? I ask because the scenario that you described (scaling in to a lower value and just instantly scaling out to the right value) usually happens because during the workload modification, the replicas value is set. When you modified the parameters, did you apply also the workload manifest (deployment, statefulset, etc) or just modify the ScaledObject? In case of applying all the files, does your workload manifest have the value replicas defined?

[rickbrouwer]

The proposal is later changed to a fallback behavior:

fallback:
    failureThreshold: 3
    replicas: 10
    behavior: <options>

Where the options are now:

behavior: “useCurrentReplicasAsMinimum”

or

behavior: “static”

So, it is a minimum. If the fallback.replicas is higher, that number will be used.
This is only useful if you want to make sure that it never scales down. So it either stays the same (current replicas) or higher (fallback replicas).

What might also be important in this discussion, but only as an addition as root cause to the initial problem.
As I see it, is that corradomatt's adjustment immediately puts it in the doFallbackScaling. At this piece of code, he immediately falls into a fallback and does nothing with the failureThreshold (that is also why increasing this value did not help him). If KEDA does have to take the failureThreshold into account at this place, a check here on the NumberOfFailures might also be a quick fix.