-
Notifications
You must be signed in to change notification settings - Fork 90
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory leak on Interceptor #1078
Comments
@JorTurFer / @wozniakjan do you have bandwidth to check this one? |
thank you for reporting this, from the graphs this indeed looks like a memory leak, I can investigate later today. |
after a quick investigation, this might actually be related to HTTP connections overhead, GC and request caching rather than an actual memory leak. I am able to reproduce quick memory growth with concurrent requests. When idle on a tiny cluster with a single
with requests sent sequentially, the memory usage barely increases and then goes down to idle levels. With simple benchmark using
the memory usage does increase, peaking at
but after Attached is also trace output from |
I may have noticed something, with higher error rate in the benchmark, the memory consumption doesn't go down. I just bumped the memory limit for interceptor to 1Gi and executed benchmark that may have been too much, 5000 connections in 200 threads for 10 minutes
out of ~350000 requests, 91% timeouts and 4.7% errors. The memory usage is still at |
@wozniakjan my current number is around 15k requests every minute. I currently have 3 interceptor pods, at 640 MB each... in about one hour and half that number is expected to peak at 25k... during the late night, between 2am-5am, the number drops to 5k... |
@Leonardo-Ferreira I played around with benchmarks a bit more and afaict With a low error rate, the memory usage appears to go down to idle levels eventually, but when the error rate is high, even when later the Go's pprof heap analysis doesn't point to any code that would accumulate memory and not release it, but also heap as counted by pprof appears to be just a fraction of memory used compared to cgroup accounting. The pprof recorded 44MB while cgroup show 318MB used. |
@wozniakjan after looking deeply at my data, I could not correlate the memory usage/increase-rate to the number of errors. Luckily (or not) we had an issue this morning where a dependency was flaky so there was a significant spike in errors related to timeouts. on the memory consumption graph there was no significant change during the 1 hour long event... |
good to hear that, perhaps my artificial benchmark setup just ends up being CPU-intensive. my current hypothesis I am trying to either prove or disprove is that maybe when the |
would that output a specific log message that I could query for? |
I can see plenty of these access logs in the
despite the fact the scaled app returns always Also, I can observe these errors in the
|
should we log some extra info and release a 0.8.1 version? unfortunately my security dpto does not allow me to use "unofficial" images in production |
I'm still in the process of trying to figure out the root cause but as soon as I have some valid hypothesis to test, I can distribute a build for your testing |
any news @wozniakjan, can I help somehow? |
I didn't get very far and now I had to put it on hold temporarily, but I will get back to this soon |
hey @wozniakjan, I'd like to contribute here. would you be willing to connect for like 30-45min so you can "boost" me? this way I could make significant contributions faster |
sure, you can ping me on KEDA slack https://keda.sh/community/ |
Report
I think we have a memory leak.
![image](https://private-user-images.githubusercontent.com/5585339/344339520-86bef39d-281c-4b7c-814c-82563284697d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQyNzUyNTYsIm5iZiI6MTcyNDI3NDk1NiwicGF0aCI6Ii81NTg1MzM5LzM0NDMzOTUyMC04NmJlZjM5ZC0yODFjLTRiN2MtODE0Yy04MjU2MzI4NDY5N2QucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgyMSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MjFUMjExNTU2WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NDcyZWUxYTNjOTc0NTU4MmYyZGFmYTM1MWIwYzZmNzhhMjhlNjY0OGFhMjA2MjhlMWQwZDBjMDE2YzNmOTkzZiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.5zpKvwibw_DdlzPmFMgTFFh-IKJ_0rFgkl394WkC4h8)
![image](https://private-user-images.githubusercontent.com/5585339/344339776-6afe4d0f-179a-4665-943a-e2a8685219e1.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQyNzUyNTYsIm5iZiI6MTcyNDI3NDk1NiwicGF0aCI6Ii81NTg1MzM5LzM0NDMzOTc3Ni02YWZlNGQwZi0xNzlhLTQ2NjUtOTQzYS1lMmE4Njg1MjE5ZTEucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgyMSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MjFUMjExNTU2WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ODY5NjY2ODRkOTIxYTk4M2Y0MzMxOTY0NTRlYWQ5YjhmZDBiM2UyMTkxMzZmOGM0NmRlNTU1OGEwODk2MzZhNiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.Zs0tIKBVdiKYivMnTr-ozGHeCz9uFhaXEt0uCgkJ7Xo)
Here is the memory usage over 2 days:
The memory drops is the pod getting OOM killed. At first we thought 64mb is too little, so we decided to experiment higher values in the last 24hrs, but the symptom persisted:
Expected Behavior
the pods should release memory and as traffic fluctuates down and not get OOM killed
Actual Behavior
Memory usage never decreases
Steps to Reproduce the Problem
Logs from KEDA HTTP operator
HTTP Add-on Version
0.8.0
Kubernetes Version
< 1.28
Platform
Microsoft Azure
Anything else?
AKS v 1.27.9
The text was updated successfully, but these errors were encountered: