Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panic during master startup #16573

Closed
smarterclayton opened this issue Sep 26, 2017 · 4 comments · Fixed by #16617
Closed

Panic during master startup #16573

smarterclayton opened this issue Sep 26, 2017 · 4 comments · Fixed by #16617
Assignees
Labels
component/kubernetes kind/bug Categorizes issue or PR as related to a bug. priority/P1 sig/master

Comments

@smarterclayton
Copy link
Contributor

On a local openshift start master branch:

E0926 18:15:27.158224   41901 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/Volumes/development/projects/origin/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/Volumes/development/projects/origin/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/Volumes/development/projects/origin/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/Cellar/go/1.9/libexec/src/runtime/asm_amd64.s:509
/usr/local/Cellar/go/1.9/libexec/src/runtime/panic.go:491
/usr/local/Cellar/go/1.9/libexec/src/runtime/panic.go:63
/usr/local/Cellar/go/1.9/libexec/src/runtime/signal_unix.go:367
/Volumes/development/projects/origin/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/node/taint_controller.go:288
/Volumes/development/projects/origin/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/node/nodecontroller.go:385
/Volumes/development/projects/origin/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/node/controller_utils.go:329
/Volumes/development/projects/origin/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/controller.go:192
<autogenerated>:1
/Volumes/development/projects/origin/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/shared_informer.go:547
/usr/local/Cellar/go/1.9/libexec/src/runtime/asm_amd64.s:2337
I0926 18:15:27.258365   41901 controller_utils.go:1032] Caches are synced for node controller
I0926 18:15:27.258417   41901 taint_controller.go:182] Starting NoExecuteTaintManager
I0926 18:15:27.258469   41901 nodecontroller.go:542] Initializing eviction metric for zone:

Line of code is

tc.nodeUpdateQueue.Add(updateItemInterface(updateItem))

Is this a data race?

@smarterclayton
Copy link
Contributor Author

@openshift/sig-master

@ironcladlou
Copy link
Contributor

Looks like kubernetes/kubernetes@928ac58 (see also: kubernetes/kubernetes#51492)

@pweil- pweil- added component/kubernetes kind/bug Categorizes issue or PR as related to a bug. priority/P1 sig/master labels Sep 28, 2017
@deads2k
Copy link
Contributor

deads2k commented Sep 29, 2017

Looks like kubernetes/kubernetes@928ac58 (see also: kubernetes/kubernetes#51492)

Actually fixed here: kubernetes/kubernetes#51633 ?

@ironcladlou
Copy link
Contributor

@deads2k

Looks like kubernetes/kubernetes@928ac58 (see also: kubernetes/kubernetes#51492)

Probably want both. kubernetes/kubernetes@928ac58 would fix the specific observed panic if the GC started the informers too soon, and kubernetes/kubernetes#51633 would close any remaining gaps.

openshift-merge-robot added a commit that referenced this issue Oct 6, 2017
Automatic merge from submit-queue (batch tested with PRs 16617, 16719).

UPSTREAM: 51633: update GC controller to wait until controllers have …

…been initialized

fixes #16573

@ironcladlou it did not pick cleanly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/kubernetes kind/bug Categorizes issue or PR as related to a bug. priority/P1 sig/master
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants