Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sentry: nil panic in pkg/sdn/plugin/pod.go", line 228, in UpdateLocalMulticastRules #13742

Closed
ncdc opened this issue Apr 12, 2017 · 5 comments
Closed

sentry: nil panic in pkg/sdn/plugin/pod.go", line 228, in UpdateLocalMulticastRules #13742

ncdc opened this issue Apr 12, 2017 · 5 comments
Assignees
@dcbw
Labels
component/networking

Comments

@ncdc
Copy link
Contributor

ncdc commented Apr 12, 2017 •
edited
Loading 

runtime.errorString: runtime error: invalid memory address or nil pointer dereference
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/cmd/util/serviceability/panic.go", line 28, in CapturePanic-fm
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go", line 51, in HandleCrash
  File "/usr/lib/golang/src/runtime/asm_amd64.s", line 479, in call32
  File "/usr/lib/golang/src/runtime/panic.go", line 458, in gopanic
  File "/usr/lib/golang/src/runtime/panic.go", line 62, in panicmem
  File "/usr/lib/golang/src/runtime/sigpanic_unix.go", line 24, in sigpanic
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/pod.go", line 228, in UpdateLocalMulticastRules
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/multitenant.go", line 100, in updatePodNetwork
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/multitenant.go", line 105, in AddNetNamespace
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/vnids_node.go", line 194, in func1
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/eventqueue.go", line 108, in func1
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go", line 420, in Pop
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/eventqueue.go", line 114, in Pop
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/common.go", line 146, in RunEventQueue
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/vnids_node.go", line 203, in watchNetNamespaces
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/vnids_node.go", line 173, in watchNetNamespaces)-fm
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go", line 96, in func1
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go", line 97, in JitterUntil
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go", line 52, in Until
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go", line 43, in Forever
  File "/usr/lib/golang/src/runtime/asm_amd64.s", line 2086, in goexit
@danwinship
Copy link
Contributor

did this happen at startup?

@ncdc
Copy link
Contributor Author

ncdc commented Apr 12, 2017

@danwinship it's hard to tell. This is all the info I have from Sentry. But it looks like it was popping off the event queue, so no?

@dcbw
Copy link
Contributor

dcbw commented Apr 12, 2017

I'm on it. But yes it would have to happen at startup.

@dcbw dcbw assigned dcbw and unassigned knobunc Apr 12, 2017
@dcbw
Copy link
Contributor

dcbw commented Apr 12, 2017

Basically, the PodManager may not be created yet by the time some of the multitenant VNID-related goroutines have started. They will attempt to access node.pm which can be nil at that point in time at startup.

The PodManager can only be created after kubelet has started up, and after we have received our local subnet CIDR hostsubnet for the node. But before that happens, we could receive other HostSubnet events for other nodes in the cluster, which would trigger a UpdateLocalMulticastRules() call, which will then crash on a missing podManager.

dcbw added a commit to dcbw/origin that referenced this issue Apr 13, 2017
@dcbw
sdn: fix initialization order to prevent crash on node startup
14f15af 
OsdnNode.Start()
   (node.pm == nil at this point)
   -> node.policy.Start()  (which is multitenant policy)
   -> mp.vnids.Start()
   -> go vmap.watchNetNamespaces()
   -> (net namespace event happens)
   -> watchNetNamespaces()
   -> vmap.policy.AddNetNamespace() (policy is multitenant)
   -> mp.updatePodNetwork()
   -> mp.node.podManager.UpdateLocalMulticastRules() (and podManager is still nil)

Create the PodManager earlier so it's not nil if we get early events.

Fixes: openshift#13742
@dcbw dcbw mentioned this issue Apr 13, 2017
Merged
dcbw added a commit to dcbw/origin that referenced this issue Apr 13, 2017
@dcbw
sdn: fix initialization order to prevent crash on node startup
57a7167 
OsdnNode.Start()
   (node.pm == nil at this point)
   -> node.policy.Start()  (which is multitenant policy)
   -> mp.vnids.Start()
   -> go vmap.watchNetNamespaces()
   -> (net namespace event happens)
   -> watchNetNamespaces()
   -> vmap.policy.AddNetNamespace() (policy is multitenant)
   -> mp.updatePodNetwork()
   -> mp.node.podManager.UpdateLocalMulticastRules() (and podManager is still nil)

Create the PodManager earlier so it's not nil if we get early events.

Fixes: openshift#13742
dcbw added a commit to dcbw/origin that referenced this issue Apr 13, 2017
@dcbw
sdn: fix initialization order to prevent crash on node startup
fc95b86 
OsdnNode.Start()
   (node.pm == nil at this point)
   -> node.policy.Start()  (which is multitenant policy)
   -> mp.vnids.Start()
   -> go vmap.watchNetNamespaces()
   -> (net namespace event happens)
   -> watchNetNamespaces()
   -> vmap.policy.AddNetNamespace() (policy is multitenant)
   -> mp.updatePodNetwork()
   -> mp.node.podManager.UpdateLocalMulticastRules() (and podManager is still nil)

Create the PodManager earlier so it's not nil if we get early events.

Fixes: openshift#13742
dcbw added a commit to dcbw/origin that referenced this issue Apr 14, 2017
@dcbw
sdn: fix initialization order to prevent crash on node startup
84502cb 
OsdnNode.Start()
   (node.pm == nil at this point)
   -> node.policy.Start()  (which is multitenant policy)
   -> mp.vnids.Start()
   -> go vmap.watchNetNamespaces()
   -> (net namespace event happens)
   -> watchNetNamespaces()
   -> vmap.policy.AddNetNamespace() (policy is multitenant)
   -> mp.updatePodNetwork()
   -> mp.node.podManager.UpdateLocalMulticastRules() (and podManager is still nil)

Create the PodManager earlier so it's not nil if we get early events.

Fixes: openshift#13742
(cherry picked from commit fc95b86)
@dcbw dcbw mentioned this issue Apr 14, 2017
Merged
dcbw added a commit to dcbw/origin that referenced this issue Apr 18, 2017
@dcbw
sdn: fix initialization order to prevent crash on node startup
9b5f066 
OsdnNode.Start()
   (node.pm == nil at this point)
   -> node.policy.Start()  (which is multitenant policy)
   -> mp.vnids.Start()
   -> go vmap.watchNetNamespaces()
   -> (net namespace event happens)
   -> watchNetNamespaces()
   -> vmap.policy.AddNetNamespace() (policy is multitenant)
   -> mp.updatePodNetwork()
   -> mp.node.podManager.UpdateLocalMulticastRules() (and podManager is still nil)

Create the PodManager earlier so it's not nil if we get early events.

Fixes: openshift#13742
dcbw added a commit to dcbw/origin that referenced this issue Apr 18, 2017
@dcbw
sdn: fix initialization order to prevent crash on node startup
b60f14a 
OsdnNode.Start()
   (node.pm == nil at this point)
   -> node.policy.Start()  (which is multitenant policy)
   -> mp.vnids.Start()
   -> go vmap.watchNetNamespaces()
   -> (net namespace event happens)
   -> watchNetNamespaces()
   -> vmap.policy.AddNetNamespace() (policy is multitenant)
   -> mp.updatePodNetwork()
   -> mp.node.podManager.UpdateLocalMulticastRules() (and podManager is still nil)

Create the PodManager earlier so it's not nil if we get early events.

Fixes: openshift#13742
dcbw added a commit to dcbw/origin that referenced this issue Apr 18, 2017
@dcbw
sdn: fix initialization order to prevent crash on node startup
0e01ff0 
OsdnNode.Start()
   (node.pm == nil at this point)
   -> node.policy.Start()  (which is multitenant policy)
   -> mp.vnids.Start()
   -> go vmap.watchNetNamespaces()
   -> (net namespace event happens)
   -> watchNetNamespaces()
   -> vmap.policy.AddNetNamespace() (policy is multitenant)
   -> mp.updatePodNetwork()
   -> mp.node.podManager.UpdateLocalMulticastRules() (and podManager is still nil)

Create the PodManager earlier so it's not nil if we get early events.

Fixes: openshift#13742
dcbw added a commit to dcbw/origin that referenced this issue Apr 19, 2017
@dcbw
sdn: fix initialization order to prevent crash on node startup
94fb0f4 
OsdnNode.Start()
   (node.pm == nil at this point)
   -> node.policy.Start()  (which is multitenant policy)
   -> mp.vnids.Start()
   -> go vmap.watchNetNamespaces()
   -> (net namespace event happens)
   -> watchNetNamespaces()
   -> vmap.policy.AddNetNamespace() (policy is multitenant)
   -> mp.updatePodNetwork()
   -> mp.node.podManager.UpdateLocalMulticastRules() (and podManager is still nil)

Create the PodManager earlier so it's not nil if we get early events.

Fixes: openshift#13742
dcbw added a commit to dcbw/origin that referenced this issue Apr 20, 2017
@dcbw
sdn: fix initialization order to prevent crash on node startup
521c79f 
OsdnNode.Start()
   (node.pm == nil at this point)
   -> node.policy.Start()  (which is multitenant policy)
   -> mp.vnids.Start()
   -> go vmap.watchNetNamespaces()
   -> (net namespace event happens)
   -> watchNetNamespaces()
   -> vmap.policy.AddNetNamespace() (policy is multitenant)
   -> mp.updatePodNetwork()
   -> mp.node.podManager.UpdateLocalMulticastRules() (and podManager is still nil)

Create the PodManager earlier so it's not nil if we get early events.

Fixes: openshift#13742
dcbw added a commit to dcbw/origin that referenced this issue Apr 20, 2017
@dcbw
sdn: fix initialization order to prevent crash on node startup
030e3ec 
OsdnNode.Start()
   (node.pm == nil at this point)
   -> node.policy.Start()  (which is multitenant policy)
   -> mp.vnids.Start()
   -> go vmap.watchNetNamespaces()
   -> (net namespace event happens)
   -> watchNetNamespaces()
   -> vmap.policy.AddNetNamespace() (policy is multitenant)
   -> mp.updatePodNetwork()
   -> mp.node.podManager.UpdateLocalMulticastRules() (and podManager is still nil)

Create the PodManager earlier so it's not nil if we get early events.

Fixes: openshift#13742
@danwinship danwinship mentioned this issue Apr 25, 2017
Closed
@knobunc
Copy link
Contributor

knobunc commented Apr 25, 2017

Bug https://bugzilla.redhat.com/show_bug.cgi?id=1445504

arilivigni pushed a commit to arilivigni/origin that referenced this issue May 4, 2017
@dcbw
sdn: fix initialization order to prevent crash on node startup
0e097cb 
OsdnNode.Start()
   (node.pm == nil at this point)
   -> node.policy.Start()  (which is multitenant policy)
   -> mp.vnids.Start()
   -> go vmap.watchNetNamespaces()
   -> (net namespace event happens)
   -> watchNetNamespaces()
   -> vmap.policy.AddNetNamespace() (policy is multitenant)
   -> mp.updatePodNetwork()
   -> mp.node.podManager.UpdateLocalMulticastRules() (and podManager is still nil)

Create the PodManager earlier so it's not nil if we get early events.

Fixes: openshift#13742
deads2k pushed a commit to openshift/sdn that referenced this issue Jun 18, 2019
@dcbw @deads2k
sdn: fix initialization order to prevent crash on node startup
63e537a 
OsdnNode.Start()
   (node.pm == nil at this point)
   -> node.policy.Start()  (which is multitenant policy)
   -> mp.vnids.Start()
   -> go vmap.watchNetNamespaces()
   -> (net namespace event happens)
   -> watchNetNamespaces()
   -> vmap.policy.AddNetNamespace() (policy is multitenant)
   -> mp.updatePodNetwork()
   -> mp.node.podManager.UpdateLocalMulticastRules() (and podManager is still nil)

Create the PodManager earlier so it's not nil if we get early events.

Fixes: openshift/origin#13742
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dcbw dcbw

Labels
component/networking
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ncdc @danwinship @knobunc @dcbw