Memory increasing linearly and finally pod crashes with OOOM

[cerberus20]

I tried configuring 2GB, than 4GB and finally 16GB of memory for controller pod but pod memory keeps increasing.
Cores allocated to container: 2

I am performance testing the controller with 100 concurrent up and downscale jobs for 100 different deployments.

apiVersion: autoscaling.alibabacloud.com/v1beta1 kind: CronHorizontalPodAutoscaler metadata: name: cronhpa-1 namespace: test spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: deployment-1 jobs: - name: "scale-down" schedule: "* */9 * * * *" targetSize: 1 - name: "scale-up" schedule: "* */7 * * * *" targetSize: 2

Can you please guide what may be the issue behind this? Thank you

[cerberus20]

@ringtail Would you be able to help here please? Looks like there is some memory leak happening. I changed the schedule to 100 jobs per minute every 9th and 7th minute, but still memory keeps increasing. Thank you.

[ringtail]

100 CronHPA Jobs and 2 times changes in one minute?

[cerberus20]

@ringtail I mean I had changed the 100 scale down jobs to 0 */9 * * * * and 100 scale up jobs to 0 */7 * * * * schedule, but still memory keeps increasing.

[cerberus20]

This is the 1 of 100 cronhpa objects:

apiVersion: autoscaling.alibabacloud.com/v1beta1
kind: CronHorizontalPodAutoscaler
metadata:
  name: cronhpa-99
  namespace: test
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: deployment-99
  jobs:
    - name: "scale-down"
      schedule: "0 */9 * * * *"
      targetSize: 1
    - name: "scale-up"
      schedule: "0 */7 * * * *"
      targetSize: 2```

[IrisIris]

@cerberus20 I guess the picture shows the memory consumption of CronHPA controller, right? CronHPA jobs are configured to execute in minutes，but the cycle time in the picture is in "day"?

[cerberus20]

@IrisIris Yes, the memory consumption is for CronHPA controller. Yeah, that picture is for the case when I had configured 100 cronhpa objects of:

apiVersion: autoscaling.alibabacloud.com/v1beta1
kind: CronHorizontalPodAutoscaler
metadata:
  name: cronhpa-{0..99}
  namespace: test
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: deployment-{0..99}
  jobs:
    - name: "scale-down"
      schedule: "* */9 * * * *"
      targetSize: 1
    - name: "scale-up"
      schedule: "* */7 * * * *"
      targetSize: 2

But even with reduced frequency jobs mentioned in last comment, memory keeps increasing but with slower rate than what is shown in picture.

Cycle is 1 day because I had configured 16 GB memory limit and controller container used to hit the limit in ~1 day.

[ringtail]

@IrisIris Does latest version still have the same problem.

[cerberus20]

I am not sure if #124 is the fix or not for the above issue, but changes with this PR as well still face the issue. cc: @ringtail @IrisIris

[ringtail]

@cerberus20 Have you try the latest version?

[cerberus20]

@ringtail Yes, I am generating controller binary from latest state of master.