File descriptor leak detected

[patagona-anas]

Describe the bug
The numbers for the open files increase constantly for the process dkron/dkron-executor-http. The resource (= file descriptor) might be released but it takes a bit long ( see Additional context for details)

To Reproduce
Steps to reproduce the behavior:

1. Create 100 jobs that execute after every 6 minutes ( executor = HTTP)
2. Check the output of the lsof | wc -l before and after the execution.
3. The open file value increases at every execution

Expected behavior
It should release the file descriptor a bit fast

Screenshots

• Sample 1 when no jobs are executed
• Sample 2-4 every 6 minutes interval

Specifications:

• OS: amazon linux
• Docker image: dkron/dkron:v2.0.0

Additional context

• Soft/hard limit inside docker container: 1024/4096
• Maybe the resources are released but it is taking too long. As a result, the job failed because of the socket: too many open files

[patagona-anas]

Bottom line

• There is no bug and the open file descriptor is not increasing constantly but getting freed over time.
• The used limit was too low and it has been increased.

X-Axis: Datetime
Y-Axis: Open file descriptor

[yvanoers]

@patagona-anas
Even so, it is still remarkable that the FDs remain open longer than we'd expect. If I get the time I'll see if I can pinpoint what those descriptors are used for, because I was looking at the code and the httpExecutor is closing all its handles as far as I've seen. Maybe there's something going on with the module system.

[vcastellm]

Thanks for raising this @patagona-anas, great investigation here

[patagona-anas]

@Victorcoder: thanks. I am then re-opening it

[vcastellm]

FTR my tests behave the same, it grows until a certain point where it remains stable. Not causing any issues at all.

[vcastellm]

Related to #904 and #1062