Confusing (wrong) description on supported replica of metrics server deployment

[zqzten]

In High Availability section of the operate doc, it said that KEDA metrics server does not support HA according to limitation in k8s custom metrics server.

However, the limitation mentioned is that k8s does not support more than one type of custom metrics server but not more than one replica of the same custom metrics server. So actually there's no limitation on the replica of KEDA metrics server and it does support HA. Seems the current doc is wrong there.

[tomkerkhove]

There is though, only 1 replica will be active and used so it does not provide HA

[zqzten]

But the KEDA metrics server is an aggregated apiserver (which naturally support HA) right? Say I have two replicas of it, and there's one down, the other one can still handle the request from HPA controller. I haven't look through the code so I'm not sure if I missed sth.

[tomkerkhove]

Yes but it's not going to actively load balance across multiple instances

[tomkerkhove]

But @zroubalik might correct me, but I'm pretty sure that's correct

[zroubalik]

Yeah, that the usual default way it works, ie. only one instance handles the load. For the requests to be balanced between all the instances of the metrics-apiserver, you need to set --enable-aggregator-routing=true on the kube-apiserver.

[zroubalik]

The issue is, that once we introduce metrics values caching on the metric server side, then we will have to somehow deal with sharing the cache between multiple replicas

[zqzten]

The issue is, that once we introduce metrics values caching on the metric server side, then we will have to somehow deal with sharing the cache between multiple replicas

Yeah that might be a problem. So the question can become: if I setup KEDA metrics server with HA (more than 1 replica), what possible issue would I meet?

[tomkerkhove]

That the only "HA" that you get, is that it will reduce failing over ones the primary fails

[zroubalik]

@zqzten no issues at the moment when specifying multiple replicas.

[zqzten]

Thanks! So shall we correct the doc to say that KEDA metrics server does support HA with multiple relicas?

[zroubalik]

Yeah, that would be great. But we should be careful with the wording, so people don't have the impression that they could have mutliple installations per cluster.

[tomkerkhove]

I would prefer not to change it given it's not really HA IMO but if everything things it's best then that's that.

What about this?

While you can run more replicas of our metric server, only one instance will used and serve traffic.

You can run multiple replicas, but they will not improve the performance of KEDA, it could only reduce downtime during a failover.

[zroubalik]

LGTM

[zqzten]

While you can run more replicas of our metric server, only one instance will used and serve traffic.

You can run multiple replicas, but they will not improve the performance of KEDA, it could only reduce downtime during a failover.

That might not be accurate enough? According to what @zroubalik mentioned, we can achive load balanced HA of KEDA metrics server by setting --enable-aggregator-routing=true on the kube-apiserver.

[zqzten]

Just found that the official metrics server also has this recommandation, FYI: https://github.com/kubernetes-sigs/metrics-server#high-availability

[zroubalik]

Yeah, we can also mention this property there. Are you willing to open a PR to fix this?

[zqzten]

Yeah, we can also mention this property there. Are you willing to open a PR to fix this?

Glad to. I'll open one in these days.