Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leak: http connections keep opened #1062

Closed
arvatoth-vcc opened this issue Feb 28, 2022 · 8 comments
Closed

Memory leak: http connections keep opened #1062

arvatoth-vcc opened this issue Feb 28, 2022 · 8 comments

Comments

@arvatoth-vcc
Copy link

Describe the bug
We have got a dkron-3.1.10 inside Kubernetes cluster with single deployment (single pod only). We are using only one http executor which started at every 30 seconds. Every check opens a new http connection and all of previous keep established.

Expected behavior
Close previous established tcp connections or re-use them and do not open new to same endpoint.

Screenshots
image

** Specifications:**

  • OS: Kubernetes v1.21.9 under Ubuntu 18.04 Bionic
  • Version dkron-3.1.0

Additional context
Lot of established tcp connections:

image

Running inside the pod:

image

Thank you!

@vcastellm
Copy link
Member

Under my observations the number of open connections with hundreds of http jobs, the connections reach a stable point. How many jobs are you trying to run?

@arvatoth-vcc
Copy link
Author

Only one job with one http executor. I think http executor sends Connection: keep-alive HTTP requests, but not reuse that.

@wuxingzhong
Copy link

I ran into the same problem where dkron would not release the connection causing server memory to overflow

@vcastellm
Copy link
Member

Can you confirm if this still happens in latest version?

@arvatoth-vcc
Copy link
Author

I have just refreshed version to 3.2.2, I can refer in a few days.

@arvatoth-vcc
Copy link
Author

image

Unfortunately version upgrade did not solve the memleak problem.

@wuxingzhong
Copy link

I set idletimeout on the server side to block this problem for now. I think this is a bug in dkron

image

@vcastellm
Copy link
Member

Closing this because the workaround was merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants