Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

task runner: fix goroutine leak in prestart hook #11741

Merged
merged 2 commits into from
Dec 23, 2021
Merged

task runner: fix goroutine leak in prestart hook #11741

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e1bbf30
task runner: fix goroutine leak in prestart hook
tgross Dec 22, 2021
7406e6d
move joincontext to top of loop
tgross Dec 23, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 3 additions & 0 deletions .changelog/11741.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
```release-note:bug
client: Fixed a memory and goroutine leak for batch tasks and any task that exits without being shut down from the server
```
6 changes: 4 additions & 2 deletions client/allocrunner/taskrunner/task_runner_hooks.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -235,9 +235,11 @@ func (tr *TaskRunner) prestart() error {
}

// Run the prestart hook
// use a joint context to allow any blocking pre-start hooks
// use a join context to allow any blocking pre-start hooks
// to be canceled by either killCtx or shutdownCtx
joinedCtx, _ := joincontext.Join(tr.killCtx, tr.shutdownCtx)
joinedCtx, joinedCancel := joincontext.Join(tr.killCtx, tr.shutdownCtx)
defer joinedCancel()
Copy link
Contributor

@notnoop notnoop Dec 23, 2021 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is a new joinedCtx per hook necessary? We can reuse the same one for all prestart hooks.

Also, I haven't touched this code in so long - would this accidentally risk stopping goroutines that launched by prestart hooks if they rely on the passed context? If we use a single joinedCtx, we can save it and cancel it in exited/stop hooks.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is a new joinedCtx per hook necessary? We can reuse the same one for all prestart hooks.

Oh, that's a good point. We could move this up to the top of the loop (but we can't pass it into this prestart method from the caller in task_runner.go because then we don't get to use the defer). Will do that.

would this accidentally risk stopping goroutines that launched by prestart hooks if they rely on the passed context?

Yes, but I went through and verified we're not doing that currently (as @lgfa29 has noted most of the prestart hooks don't even use the context). I feel fairly confident saying that saving the context in a goroutine in a prestart hook is not a behavior we should be doing at all, given how we treat prestart hooks? I can add a note to warn future developers about it, at least.


var resp interfaces.TaskPrestartResponse
if err := pre.Prestart(joinedCtx, &req, &resp); err != nil {
tr.emitHookError(err, name)
Expand Down