CSI: set mounts in alloc hook resources atomically #16722
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tgross
added
type/bug
theme/storage
backport/1.3.x
The allocrunner has a facility for passing data written by allocrunner hooks to taskrunner hooks. Currently the only consumers of this facility are the allocrunner CSI hook (which writes data) and the taskrunner volume hook (which reads that same data). The allocrunner hook for CSI volumes doesn't set the alloc hook resources atomically. Instead, it gets the current resources and then writes a new version back. Because the CSI hook is currently the only writer and all readers happen long afterwards, this should be safe but #16623 shows there's some sequence of events during restore where this breaks down. Refactor hook resources so that hook data is accessed via setters and getters that hold the mutex.
tgross
force-pushed
the
alloc-hook-resources
branch
from
2d84eef
to
ee11097
Compare
shoenig
approved these changes
Apr 3, 2023
client/structs/allochook.go
Outdated
Comment on lines
25
to
30
| func (a *AllocHookResources) GetCSIMounts() map[string]*csimanager.MountInfo { | ||
| a.mu.RLock() | ||
| defer a.mu.RUnlock() | ||
|
|
||
| return a.CSIMounts | ||
| return a.csiMounts | ||
| } |
There was a problem hiding this comment.
What's the policy on modifying this map? Maybe add a comment here on what is expected of callers, or return a copy.
There was a problem hiding this comment.
Good call. It's probably safe to "guard" this with a comment because the volume hook is just reading that data and not writing it, but it's way too easy to break later unexpectedly. Returning a copy in a9b2ac8
|
This will go out in the next regular patch release. |
This was referenced Apr 3, 2023
tgross
added a commit
that referenced
this pull request
Apr 3, 2023
The allocrunner has a facility for passing data written by allocrunner hooks to taskrunner hooks. Currently the only consumers of this facility are the allocrunner CSI hook (which writes data) and the taskrunner volume hook (which reads that same data). The allocrunner hook for CSI volumes doesn't set the alloc hook resources atomically. Instead, it gets the current resources and then writes a new version back. Because the CSI hook is currently the only writer and all readers happen long afterwards, this should be safe but #16623 shows there's some sequence of events during restore where this breaks down. Refactor hook resources so that hook data is accessed via setters and getters that hold the mutex.
tgross
added a commit
that referenced
this pull request
Apr 3, 2023
The allocrunner has a facility for passing data written by allocrunner hooks to taskrunner hooks. Currently the only consumers of this facility are the allocrunner CSI hook (which writes data) and the taskrunner volume hook (which reads that same data). The allocrunner hook for CSI volumes doesn't set the alloc hook resources atomically. Instead, it gets the current resources and then writes a new version back. Because the CSI hook is currently the only writer and all readers happen long afterwards, this should be safe but #16623 shows there's some sequence of events during restore where this breaks down. Refactor hook resources so that hook data is accessed via setters and getters that hold the mutex. Co-authored-by: Tim Gross <tgross@hashicorp.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #16623
The allocrunner has a facility for passing data written by allocrunner hooks to taskrunner hooks. Currently the only consumers of this facility are the allocrunner CSI hook (which writes data) and the taskrunner volume hook (which reads that same data).
The allocrunner hook for CSI volumes doesn't set the alloc hook resources atomically. Instead, it gets the current resources and then writes a new version back. Because the CSI hook is currently the only writer and all readers happen long afterwards, this should be safe but #16623 shows there's some sequence of events during restore where this breaks down.
Refactor hook resources so that hook data is accessed via setters and getters that hold the mutex, and ensure the object is instantiated synchronously at the time the
AllocRunneris created.Note to reviewers: the reproduction for the crash is extremely complicated and timing dependent, so we're not going to be able to test this in a unit test. See my comment at #16623 (comment) for how this has been manually tested. There's some follow-up work we're going to need to do in #16746