Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nomad Client leaks goroutines. #15179

Closed
stswidwinski opened this issue Nov 7, 2022 · 3 comments
Closed

Nomad Client leaks goroutines. #15179

stswidwinski opened this issue Nov 7, 2022 · 3 comments

Comments

@stswidwinski
Copy link
Contributor

stswidwinski commented Nov 7, 2022

Nomad version

1.4.2

Operating system and Environment details

Unix

Issue

Nomad client leaks goroutines under normal operation, at least 1 goroutine per task. The exact reason why:

  1. When one creates a task, one creates a context to await for the exit of the task and write the exit code

https://github.com/hashicorp/nomad/blob/main/plugins/drivers/client.go#L190

This context is never cancelled unless the parent contexts are cancelled or done.

  1. The parent contexts are very long lived. The plugin context has the lifetime of the plugin itself:

https://github.com/hashicorp/nomad/blob/main/plugins/drivers/client.go#L32

The other one has the lifetime of... forever

https://github.com/hashicorp/nomad/blob/main/client/allocrunner/taskrunner/task_runner.go#L623

Because none of them are ever cancelled we leak goroutines. The goroutine leaked is started here:

https://github.com/LK4D4/joincontext/blob/master/context.go#L33

Reproduction steps

See above. Run any tasks in a tight loop. It is also relatively easy to test by restarting the plugin and observing that all of the goroutines are freed as a result.

@jrasell
Copy link
Member

jrasell commented Nov 14, 2022

Hi @stswidwinski and thanks for raising this issue along with a PR. I see that Seth has assigned himself as reviewer on the PR, therefore I will assign this to him also to complete the loop.

@shoenig
Copy link
Member

shoenig commented Nov 17, 2022

Closed by #15180

@shoenig shoenig closed this as completed Nov 17, 2022
Nomad - Community Issues Triage automation moved this from In Progress to Done Nov 17, 2022
@github-actions
Copy link

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 18, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
Development

No branches or pull requests

3 participants