How to use HPAScaleToZero=true flag?

[Orfeasfil]

How can I use the HPAScaleToZero flag in karmada? I tried to edit the karmada-apiserver deployment, but still does not let me to set the minReplicas field in a fhpa.yaml file equal to 0. When I try to do this i get the following error:

Error from server (Forbidden): error when creating "fhpa-c.yml": admission webhook "federatedhpa.karmada.io" denied the request: spec.minReplicas: Invalid value: 0: must be greater than or equal to 1

Any idea here? This is a very common scenario and a guide on how to do this will be very helpful!

Environment:

• Karmada version: v1.8
• Kubernetes version: v1.27.12

[chaosi-zju]

This is a very common scenario

Hi @Orfeasfil, why you think this is a common scenario? Are you also use this way in single cluster?

As for spec.minReplicas: Invalid value: 0: must be greater than or equal to 1, if you must set spec.minReplicas to hero, refer to https://midbai.com/en/post/hpa-scale-to-zero/, you should turn on the feature gate HPAScaleToZero.

In Karmada, you can do like this:

kubectl edit deployment karmada-apiserver -n karmada-system

then modify the spec.template.spec.containers[0].command field and add the line - --feature-gates=HPAScaleToZero=true, after this the yaml will just like:

...
containers:
  - command:
    - kube-apiserver
    - --feature-gates=HPAScaleToZero=true
    ...
...  

thus, you can resolve the above error log, but you will encounter a new error spec.metrics: Forbidden: must specify at least one Object or External metric to support scaling to zero replicas. However, I don't quite known how to resolve this new error.

Maybe some help you can find in https://midbai.com/en/post/hpa-scale-to-zero/

[OrfeasfilMLAB]

When I tried to do this:

kubectl edit deployment karmada-apiserver -n karmada-system

and add the following line

- --feature-gates=HPAScaleToZero=true

here:

...
containers:
  - command:
    - kube-apiserver
    - --feature-gates=HPAScaleToZero=true
    ...
...  

then a new karmada-api-server pod is in pending state (with the new configurations):

when deleting the running pod, then the new pod with the new configurations (that should support the minReplicas equal to 0) is running and the other is in pending. When trying to apply an FHPA with minReplicas equal to 0 I get now this error:

Error from server (Forbidden): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"autoscaling.karmada.io/v1alpha1\",\"kind\":\"FederatedHPA\",\"metadata\":{\"annotations\":{},\"name\":\"nephele-monitor-g\",\"namespace\":\"nephele-monitor-test\"},\"spec\":{\"behavior\":{\"scaleDown\":{\"stabilizationWindowSeconds\":10},\"scaleUp\":{\"stabilizationWindowSeconds\":10}},\"maxReplicas\":2,\"metrics\":[{\"pods\":{\"metric\":{\"name\":\"sample_gauge\"},\"target\":{\"averageValue\":100,\"type\":\"Value\"}},\"type\":\"Pods\"}],\"minReplicas\":0,\"scaleTargetRef\":{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"name\":\"nephele-monitor-g\"}}}\n"}},"spec":{"metrics":[{"pods":{"metric":{"name":"sample_gauge"},"target":{"averageValue":100,"type":"Value"}},"type":"Pods"}],"minReplicas":0}}
to:
Resource: "autoscaling.karmada.io/v1alpha1, Resource=federatedhpas", GroupVersionKind: "autoscaling.karmada.io/v1alpha1, Kind=FederatedHPA"
Name: "nephele-monitor-g", Namespace: "nephele-monitor-test"
for: "fhpa-g.yml": error when patching "fhpa-g.yml": admission webhook "federatedhpa.karmada.io" denied the request: spec.minReplicas: Invalid value: 0: must be greater than or equal to 1

Did you encounter this error?

Did you follow any other procedure instead of trying deleting pods?

Thank you for your time

[wu0407]

The feature only supports MetricSourceType as object and external

currently in alpha status, and it only supports HPA of object and external types.

https://midbai.com/en/post/hpa-scale-to-zero/#does-kubernetes-hpa-support-scaling-down-to-0

[Orfeasfil]

Yes, thats why i have the following configuration yaml for the FHPA:

apiVersion: autoscaling.karmada.io/v1alpha1
kind: FederatedHPA
metadata:
  name: nephele-monitor-g
  namespace: nephele-monitor-test
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nephele-monitor-g
  minReplicas: 0
  maxReplicas: 2
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 10
    scaleUp:
      stabilizationWindowSeconds: 10
    metrics:
     - type: External
       external:
         metric:
           name: sample_gauge
         target:
           type: AverageValue
           averageValue: 100

where I use the External metrics API to get the metrics. The External metrics API works ok because the same thing with HPA worked fine.

[chaosi-zju]

• type: External
  external:
  metric:
  name: sample_gauge
  target:
  type: AverageValue
  averageValue: 100

The External metrics API works ok

Just curious to ask, does your External metrics API works ok in Karmada? It seems that Karmada now doesn't support External type metrics, you shall got such error: Unsupported value: "External": supported values: "ContainerResource", "Object", "Pods", "Resource"