-
Notifications
You must be signed in to change notification settings - Fork 121
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory increasing linearly and finally pod crashes with OOOM #116
Comments
@ringtail Would you be able to help here please? Looks like there is some memory leak happening. I changed the schedule to 100 jobs per minute every 9th and 7th minute, but still memory keeps increasing. Thank you. |
100 CronHPA Jobs and 2 times changes in one minute? |
@ringtail I mean I had changed the 100 scale down jobs to 0 */9 * * * * and 100 scale up jobs to 0 */7 * * * * schedule, but still memory keeps increasing. |
This is the 1 of 100 cronhpa objects:
|
@cerberus20 I guess the picture shows the memory consumption of CronHPA controller, right? CronHPA jobs are configured to execute in minutes,but the cycle time in the picture is in "day"? |
@IrisIris Yes, the memory consumption is for CronHPA controller. Yeah, that picture is for the case when I had configured 100 cronhpa objects of:
But even with reduced frequency jobs mentioned in last comment, memory keeps increasing but with slower rate than what is shown in picture. Cycle is 1 day because I had configured 16 GB memory limit and controller container used to hit the limit in ~1 day. |
@IrisIris Does latest version still have the same problem. |
@cerberus20 Have you try the latest version? |
@ringtail Yes, I am generating controller binary from latest state of master. |
I tried configuring 2GB, than 4GB and finally 16GB of memory for controller pod but pod memory keeps increasing.
Cores allocated to container: 2
I am performance testing the controller with 100 concurrent up and downscale jobs for 100 different deployments.
apiVersion: autoscaling.alibabacloud.com/v1beta1 kind: CronHorizontalPodAutoscaler metadata: name: cronhpa-1 namespace: test spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: deployment-1 jobs: - name: "scale-down" schedule: "* */9 * * * *" targetSize: 1 - name: "scale-up" schedule: "* */7 * * * *" targetSize: 2
Can you please guide what may be the issue behind this? Thank you
The text was updated successfully, but these errors were encountered: