-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🐛 set partialmetadata gvk in list/watch funcs to avoid data race in cache #1650
🐛 set partialmetadata gvk in list/watch funcs to avoid data race in cache #1650
Conversation
…n cache Signed-off-by: Joe Lanford <joe.lanford@gmail.com>
What exactly races there? |
With just the added test, the race detector shows this: Race detector details
TL;DR: it's multiple goroutines (one from each controller) trying to call SetGroupVersionKind on the same underlying object concurrently. |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: alvaroaleman, joelanford The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
func newGVKFixupWatcher(gvk schema.GroupVersionKind, watcher watch.Interface) watch.Interface { | ||
ch := make(chan watch.Event) | ||
w := &gvkFixupWatcher{ | ||
gvk: gvk, | ||
watcher: watcher, | ||
ch: ch, | ||
} | ||
w.wg.Add(1) | ||
go w.run() | ||
return w | ||
} | ||
|
||
func (w *gvkFixupWatcher) run() { | ||
for e := range w.watcher.ResultChan() { | ||
e.Object.GetObjectKind().SetGroupVersionKind(w.gvk) | ||
w.ch <- e | ||
} | ||
w.wg.Done() | ||
} | ||
|
||
func (w *gvkFixupWatcher) Stop() { | ||
w.watcher.Stop() | ||
w.wg.Wait() | ||
close(w.ch) | ||
} | ||
|
||
func (w *gvkFixupWatcher) ResultChan() <-chan watch.Event { | ||
return w.ch | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this the only way we can fix this behavior? Having an extra reader seems a bit overkill, given that's a reader on a reader, but I also don't have other options that I can think about.
When multiple controllers are watching the same resource type with a metadata-only informer, a data race occurs in
sigs.k8s.io/controller-runtime/pkg/cache/internal.(*handlerPreserveGVK).resetGroupVersionKind()
This data race is resolved by setting the GVK before the objects are written to the cache.
Signed-off-by: Joe Lanford joe.lanford@gmail.com